Detectron2: Example for rotated faster rcnn

[cjt222]

Thanks for great work, i have a question, how can i set config file to detect oriention target in faster rcnn or other network?

[ppwwyyxx]

What do you mean by "oriention target"?

If you mean objects that are often not horizontal/vertically aligned, like texts, detectron2 supports rotated bounding boxes and we'll provide some example configs later.

[cjt222]

Thanks for your reply!! I still have some question:
1、what kind of network can support rotated bounding boxes？
2、what kind of datasets have you tried when detecting with rotated bounding boxes and how to calculate mAP while eval and could you give me a baseline?
3、Could you share papers in rotated bounding boxes in detectron2?

[ppwwyyxx]

There will be more details when a complete training example is released.

[Aswantm]

State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully-convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. The code will be released.

[ibtingzon]

Hi, would like to follow up on this. Would you have a minimal working example on training and evaluating models using rotated bounding boxes? Thanks!

[freesunshine]

We almost cry out for it! A minimal working example is needed!Thanks!

[WeihongM]

@ppwwyyxx Any updates?

[edwardchaos]

I'm attempting to do this as well, so far I've found these configs:

cfg.MODEL.PROPOSAL_GENERATOR.NAME = "RRPN"
cfg.MODEL.ANCHOR_GENERATOR.NAME = "RotatedAnchorGenerator"

Currently, the angles in anchors are at 90 degree intervals so i'll need to add more angles.

I'm using the faster_rcnn_R_101_FPN_3x pretrained network.

The issue now is using detectron2's dataloader with rotated bounding box annotations.
In the documentation it says that box must be a member of detectron2.structures.BoxMode which only supports horizontal bounding box annotations.

To get it to work now I'm thinking i'll have to use another way of loading data.
@ppwwyyxx do you think it will be available in days, weeks, or months?

[ppwwyyxx]

Yes a custom dataloader is needed to provide rotated boxes as input. We don't have a concrete timeline but I think an example can be provided in a few months.

[edwardchaos]

Update:

I found another config that needs to be overridden:
MODEL.RPN.BBOX_REG_WEIGHTS: (1.0, 1.0, 1.0, 1.0, 1.0)

The default configs file doesn't have a section for RRPN, assuming it's intentional and we're supposed to override RPN configs

[freesunshine]

@edwardchaos Appreciate you for working on it! Can you keep on updating your progress here? That will be very helpful!

[edwardchaos]

Yup, now I'm taking a step back and getting horizontal bounding boxes to work first, found another config that needs to be changed for rotated boxes:

MODEL.ROI_BOX_HEAD.POOLER_TYPE = "ROIAlignRotated"

[edwardchaos]

ok I got rotated bounding box working on DOTA dataset:
https://captain-whu.github.io/DOTA/dataset.html

for training, after several epochs, it'll fail but I'm pretty sure the issue is with respect to invalid ground truth bounding boxes in my dataset.

@ppwwyyxx do you suggest I first train rrpn and then rroi heads rather than end to end?

For inference, get_deltas() calls torch.div_() which resulted in a pytorch error. I got past this by changing the in place version to torch.div(). Not sure if this is considered a bug.

I wrote a custom dataloader and data augmentation using imgaug.
Dataloader prepares the data in detectron2 input format (list of dictionaries)

As far as this issue goes, here are my configs that overwrite the base rcnn fpn config:

_BASE_: "./Base-RCNN-FPN.yaml"
MODEL:
PIXEL_MEAN: [103.530, 116.280, 123.675] # Pixel mean of imagenet dataset R/G/B?
WEIGHTS: "./model_final_a3ec72.pkl"
MASK_ON: False
RESNETS:
DEPTH: 101
PROPOSAL_GENERATOR:
NAME: "RRPN"
RPN:
HEAD_NAME: "StandardRPNHead"
BBOX_REG_WEIGHTS: (1.0, 1.0, 1.0, 1.0, 1.0)
ANCHOR_GENERATOR:
NAME: "RotatedAnchorGenerator"
ANGLES: [[0, 30, 60, 90]]
ASPECT_RATIOS: [[0.5, 1.0, 2.0]]
ROI_HEADS:
NAME: "RROIHeads"
NUM_CLASSES: 15 # number of foreground classes
ROI_BOX_HEAD:
POOLER_TYPE: "ROIAlignRotated"
BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0, 10.0)
SOLVER:
STEPS: (210000, 250000)
MAX_ITER: 10 # number of epochs ( I modified the plain net train from iterations to epochs)
CHECKPOINT_PERIOD: 1 # number of epochs

[WeihongM]

@edwardchaos Hello, how do you prepare your data loader, can you share the script here?
Also, how you prepare the angle theta.
Thanks.

[MaximKuklin]

Hi @edwardchaos, many thanks for your config! I faced the same problem with torch.div_() func. So, this can be fixed by putting torch.no_grad(). This helps because we are working with self.weights field, which contains gradient.

with torch.no_grad():
    wx, wy, ww, wh, wa = self.weights
    dx, dy, dw, dh, da = torch.unbind(deltas, dim=1)

    dx.div_(wx)
    dy.div_(wy)
    dw.div_(ww)
    dh.div_(wh)
    da.div_(wa)

I found solution here: https://discuss.pytorch.org/t/runtimeerror-output-nr-0-assert-failed-at-pytorch-torch-csrc-autograd-variable-cpp-196-please-report-a-bug-to-pytorch/39604/2

Moreover, can you share the info about how long does it take to train your model and how many images in dataset do you have, please?

[edwardchaos]

@WeihongM , this should help: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html
I'm not sure I can share the code but basically you'll want to get a list of dictionaries where each dictionary corresponds to a training sample with keys "image", "gt_instance", etc. Image should bea pytorch tensor and gt_instances is an instances object. Part of debugging, you can step through the network and check the exact keys it expects. The design pattern makes stepping through the code a little confusing at first as it jumps from place to place.

Angle representation is described in the comment here. When calculating angle, you'll come across atan or atan2. Should use atan2.

@MaximKuklin , I've trained it on DOTA 1.0 train set splitting the image down to 256x256 squares and removing any samples that don't have anything objects in them. Hardware is 2080Ti and 8 core cpu. Takes about a day or 2 for ~200k iterations (minibatch size 1). I'd train it for a week total; it depends on your dataset, and learning configurations.

I've also adapted this rotated faster rcnn to SCRDet and found it to perform better.

[kurapan]

Are rotated bounding boxes available for Mask R-CNN too?

[zhenglilei]

Looking forward to the example in 2019. 😄

[Kongsea]

Look forward to it too.

[kakaluote]

hello @edwardchaos i use your config, and write my dataloader, i can start to train r2cnn, but the train crashed after several step, CUDA out of memory

[02/22 19:08:27 d2.utils.events]: eta: 0:53:32 iter: 16 total_loss: 2.584 loss_cls: 0.722 loss_box_reg: 0.004 loss_rpn_cls: 0.694 loss_rpn_loc: 1.209 time: 0.4244 data_time: 0.0099 lr: 0.000017 max_mem: 3168M
[02/22 19:08:27 d2.utils.events]: eta: 0:54:36 iter: 17 total_loss: 2.656 loss_cls: 0.709 loss_box_reg: 0.004 loss_rpn_cls: 0.694 loss_rpn_loc: 1.210 time: 0.4257 data_time: 0.0094 lr: 0.000018 max_mem: 3168M
[02/22 19:08:28 d2.utils.events]: eta: 0:53:31 iter: 18 total_loss: 2.584 loss_cls: 0.695 loss_box_reg: 0.004 loss_rpn_cls: 0.694 loss_rpn_loc: 1.211 time: 0.4230 data_time: 0.0090 lr: 0.000019 max_mem: 3168M
ERROR [02/22 19:08:28 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
File "/home/1T/workspace/detectron2/tools/../detectron2/engine/train_loop.py", line 132, in train
self.run_step()
File "/home/1T/workspace/detectron2/tools/../detectron2/engine/train_loop.py", line 214, in run_step
loss_dict = self.model(data)
File "/home/kakaluote/anaconda3/envs/pytorch-py3-lastest/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(input, *kwargs)
File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/meta_arch/rcnn.py", line 123, in forward
proposals, proposal_losses = self.proposal_generator(images, features, gt_instances)
File "/home/kakaluote/anaconda3/envs/pytorch-py3-lastest/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(input, *kwargs)
File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/proposal_generator/rrpn.py", line 53, in forward
losses = outputs.losses()
File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/proposal_generator/rpn_outputs.py", line 322, in losses
gt_objectness_logits, gt_anchor_deltas = self._get_ground_truth()
File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/proposal_generator/rrpn_outputs.py", line 219, in _get_ground_truth
matched_idxs, gt_objectness_logits_i = self.anchor_matcher(match_quality_matrix)
File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/matcher.py", line 95, in __call__
self.set_low_quality_matches_(match_labels, match_quality_matrix)
File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/matcher.py", line 115, in set_low_quality_matches_
match_quality_matrix == highest_quality_foreach_gt[:, None]
RuntimeError: CUDA out of memory. Tried to allocate 2.53 GiB (GPU 0; 7.76 GiB total capacity; 2.08 GiB already allocated; 1.21 GiB free; 1.42 GiB cached)
[02/22 19:08:28 d2.engine.hooks]: Overall training speed: 17 iterations in 0:00:07 (0.4509 s / it)
[02/22 19:08:28 d2.engine.hooks]: Total training time: 0:00:08 (0:00:00 on hooks)

Do you have any suggestions？need your help， thanks

[edwardchaos]

Try reducing your batch size or input volume or network

[kakaluote]

yes，batchsize is 1， input is （600，） ，backbone is res50fpn
same crash。。。
Could it be a crash caused by input data?
i'm sure the angle is (-180,180)

[minhajul-arifin-badhon]

Hello @ppwwyyxx . I am using these configs:

`cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("train",)
cfg.DATASETS.TEST = ("val",)

cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml") # Let training initialize from model zoo

cfg.MODEL.PROPOSAL_GENERATOR.NAME = "RRPN"

cfg.MODEL.RPN.HEAD_NAME = "StandardRPNHead"
cfg.MODEL.RPN.BBOX_REG_WEIGHTS = (1.0, 1.0, 1.0, 1.0, 1.0)

cfg.MODEL.ANCHOR_GENERATOR.NAME = "RotatedAnchorGenerator"
cfg.MODEL.ANCHOR_GENERATOR.ANGLES = [[0, 30, 60, 90]]
cfg.MODEL.ANCHOR_GENERATOR.ASPECT_RATIOS = [[0.5, 1.0, 2.0]]

cfg.MODEL.ROI_HEADS.NAME = "RROIHeads"
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128 # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1 # only has one class

cfg.MODEL.ROI_BOX_HEAD.POOLER_TYPE = "ROIAlignRotated"
cfg.MODEL.ROI_BOX_HEAD.BBOX_REG_WEIGHTS = (10.0, 10.0, 5.0, 5.0, 10.0)

cfg.SOLVER.MAX_ITER = 300 # 300 iterations seems good enough for this toy dataset; you may need to train longer for a practical dataset`

but getting the following errors after running:

`[02/24 17:55:30 d2.engine.train_loop]: Starting training from iteration 0
ERROR [02/24 17:55:33 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/detectron2/engine/train_loop.py", line 132, in train
self.run_step()
File "/usr/local/lib/python3.6/dist-packages/detectron2/engine/train_loop.py", line 214, in run_step
loss_dict = self.model(data)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(input, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/meta_arch/rcnn.py", line 123, in forward
proposals, proposal_losses = self.proposal_generator(images, features, gt_instances)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(input, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/proposal_generator/rrpn.py", line 53, in forward
losses = outputs.losses()
File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/proposal_generator/rpn_outputs.py", line 322, in losses
gt_objectness_logits, gt_anchor_deltas = self._get_ground_truth()
File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/proposal_generator/rrpn_outputs.py", line 234, in _get_ground_truth
anchors_i.tensor, matched_gt_boxes.tensor
File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/box_regression.py", line 149, in get_deltas
target_boxes, dim=1
ValueError: not enough values to unpack (expected 5, got 4)
I am not sure how to fix this. I assume I don't need a custom dataloader anymore. (not using)
Can you please help? Thanks.

[kakaluote]

update
I found that the number of objects in image is related to the memory usage
on my dataset，some image have 47 objects, and the training always crash at these images.
27 objects will use 4171M GPU memory.
more info: crash place is same as below

File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/proposal_generator/rrpn_outputs.py", line 219, in get_ground_truth
matched_idxs, gt_objectness_logits_i = self.anchor_matcher(match_quality_matrix)
File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/matcher.py", line 95, in call
self.set_low_quality_matches(match_labels, match_quality_matrix)
File "/home/1T/workspace/detectron2/tools/../detectron2/modeling/matcher.py", line 115, in set_low_quality_matches
match_quality_matrix == highest_quality_foreach_gt[:, None]
RuntimeError: CUDA out of memory. Tried to allocate 2.53 GiB (GPU 0; 7.76 GiB total capacity; 2.08 GiB already allocated; 1.21 GiB free; 1.42 GiB cached)

@ppwwyyxx Why the number of targets is related to memory？

[cbasavaraj]

I am yet to try rotated bounding boxes, but hoping to soon. Good question by @kakaluote. In the normal faster-rcnn, the training batch sizes are fixed both for the RPN and the second stage (RCNN). So even if we have more GT boxes, and hence more foreground proposals / ROIs, there should be no proportional increase in memory allocated. At least that's what I would think.

[st7ma784]

@minhajul-arifin-badhon It looks like when the model is beginning training your dataset only has 4 values where it should have 5. From what I can tell, RRCNN methods require an extra BBox value for rotation (I suspect -im about to look and play around- that its x,y,w,h, angle)
BoxMode.XYWHA_ABS is what you want?

[minhajul-arifin-badhon]

    record = {}
    record["file_name"] = filename
    record["image_id"] = idx
    record["height"] = 1024
    record["width"] = 1024

    objs = []
    for row_index, row in df_group.iterrows():
        obj = {
            "bbox": [row['x_c'], row['y_c'], row['w'], row['h'], row['a']],
            "bbox_mode": BoxMode.XYWHA_ABS,
            "category_id": 0,
            "iscrowd": 0
        }
        objs.append(obj)

    record["annotations"] = objs

@st7ma784 yes, I want to train with rotated bounding boxes. I have used the above code to get each of the annotations. I think I am using angles properly. I also tried to train with only two images and checked that the data I am providing is correct. Not sure what I am missing. Have you tried training with rotated bounding boxes?

[st7ma784]

I havent got it working yet(being optimistic here :-P ). my guess is that

has something to do with it

[Amir22010]

help me to resolve this issue,,,i am using BoxMode.XYWHA_ABS
ERROR [03/01 19:44:57 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
File "/content/detectron2_repo/detectron2/engine/train_loop.py", line 132, in train
self.run_step()
File "/content/detectron2_repo/detectron2/engine/train_loop.py", line 215, in run_step
loss_dict = self.model(data)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(input, *kwargs)
File "/content/detectron2_repo/detectron2/modeling/meta_arch/rcnn.py", line 124, in forward
proposals, proposal_losses = self.proposal_generator(images, features, gt_instances)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(input, *kwargs)
File "/content/detectron2_repo/detectron2/modeling/proposal_generator/rrpn.py", line 53, in forward
losses = outputs.losses()
File "/content/detectron2_repo/detectron2/modeling/proposal_generator/rpn_outputs.py", line 327, in losses
gt_objectness_logits, gt_anchor_deltas = self._get_ground_truth()
File "/content/detectron2_repo/detectron2/modeling/proposal_generator/rrpn_outputs.py", line 238, in _get_ground_truth
anchors_i.tensor, matched_gt_boxes.tensor
File "/content/detectron2_repo/detectron2/modeling/box_regression.py", line 157, in get_deltas
target_boxes, dim=1

ValueError: not enough values to unpack (expected 5, got 4)

ValueError Traceback (most recent call last)

in ()
132 trainer = DefaultTrainer(cfg)
133 trainer.resume_or_load(resume=False)
--> 134 trainer.train()

156         target_ctr_x, target_ctr_y, target_widths, target_heights, target_angles = torch.unbind(

--> 157 target_boxes, dim=1
158 )
159

[st7ma784]

so, interesting development.
print(DatasetCatalog.list())
DatasetCatalog.register("Train", returnTRAINRECORDS)
MetadataCatalog.get("Train").set(thing_classes=list(ClassesNames.keys()))

try:
convert_to_coco_json("Train", ''.join("./output/"+"Train"+"_coco_format.json"))
print("Succeeded in making Train_coco_format.json")
MetadataCatalog.get("Train").set(json_file=''.join("./output/"+"Train"+"_coco_format.json"))
except:
pass`

Is what Im using to load my dataset into a coco format JSON file, meaning it can be used later in an evaluator. I'm suspicious that this doesnt get to the print in the "try" section when the Bbox mode is XYWHA_ABS ... might this be the cause of only getting 4 values when resolving deltas?

[Amir22010]

with my own code to generate data dict

i am getting dataset_dicts

[{'annotations': [{'bbox': [465.1667, 696.8333, 757.0, 67.0, 0.0],
'bbox_mode': ,
'category_id': 1,
'iscrowd': 0},
{'bbox': [238.4438, 263.116, 343.7455, 38.4476, 0.02],
'bbox_mode': ,
'category_id': 0,
'iscrowd': 0}],
'file_name': '/content/gdrive/My Drive/app/detectro/images/ad7ab3c1-1e87-43e2-af06-c53ed76e9e82.jpg',
'height': 1010,
'image_id': 0,
'width': 898},
{'annotations': [{'bbox': [425.295, 612.7028, 247.3418, 25.9827, 3.021593],
'bbox_mode': ,
'category_id': 1,
'iscrowd': 0},
{'bbox': [82.0213, 480.0, 115.0, 26.0, 2.991593],
'bbox_mode': ,
'category_id': 0,
'iscrowd': 0}],
'file_name': '/content/gdrive/My Drive/app/detectro/images/fa981233-a7f2-4bb5-84b1-8f7a5cc06011.jpg',
'height': 812,
'image_id': 1,
'width': 575},

then i register with below code is it correct??????

from detectron2.data import DatasetCatalog, MetadataCatalog
for d in ["train"]:
DatasetCatalog.register("notic_" + d, lambda d=d: get_data_dicts(image_paths))
MetadataCatalog.get("notic_" + d).set(thing_classes=classes)

[breznak]

Guys, I've been working on broken RRPN and only now found this useful thread!
If you can help me review and implement needed missing piece in the PR #996 , I think we can finally make this work out of the box :card_file_box:

[ryangooch]

@breznak Thanks for your work on this! I've been trying to get this workflow working too.

I'm not sure if this is the right place to discuss (or if it should be on your PR) but I have been running into a problem with your code. The error occurs when I am trying to train (I'm using some of the above config options as well as my own data loaded):

```AssertionError Traceback (most recent call last)
in
1 os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
----> 2 trainer = DefaultTrainer(cfg)
3 trainer.resume_or_load(resume=False)

c:users\rgooch\documents\detectron2_repo\detectron2\engine\defaults.py in __init__(self, cfg)
249 setup_logger()
250 # Assume these objects must be constructed in this order.
--> 251 model = self.build_model(cfg)
252 optimizer = self.build_optimizer(cfg, model)
253 data_loader = self.build_train_loader(cfg)

c:users\rgooch\documents\detectron2_repo\detectron2\engine\defaults.py in build_model(cls, cfg)
395 Overwrite it if you'd like a different model.
396 """
--> 397 model = build_model(cfg)
398 logger = logging.getLogger(__name__)
399 logger.info("Model:\n{}".format(model))

c:users\rgooch\documents\detectron2_repo\detectron2\modeling\meta_archbuild.py in build_model(cfg)
17 """
18 meta_arch = cfg.MODEL.META_ARCHITECTURE
---> 19 return META_ARCH_REGISTRY.get(meta_arch)(cfg)

c:users\rgooch\documents\detectron2_repo\detectron2\modeling\meta_arch\rcnn.py in __init__(self, cfg)
32 self.device = torch.device(cfg.MODEL.DEVICE)
33 self.backbone = build_backbone(cfg)
---> 34 self.proposal_generator = build_proposal_generator(cfg, self.backbone.output_shape())
35 self.roi_heads = build_roi_heads(cfg, self.backbone.output_shape())
36 self.vis_period = cfg.VIS_PERIOD

c:users\rgooch\documents\detectron2_repo\detectron2\modeling\proposal_generatorbuild.py in build_proposal_generator(cfg, input_shape)
22 return None
23
---> 24 return PROPOSAL_GENERATOR_REGISTRY.get(name)(cfg, input_shape)

c:users\rgooch\documents\detectron2_repo\detectron2\modeling\proposal_generator\rrpn.py in __init__(self, cfg, input_shape)
26
27 def __init__(self, cfg, input_shape: Dict[str, ShapeSpec]):
---> 28 super(RRPN, self).__init__(cfg, input_shape)
29 # three lines below added for pull request 996
30 assert(isinstance(self.anchor_generator, RotatedAnchorGenerator)), \

c:users\rgooch\documents\detectron2_repo\detectron2\modeling\proposal_generator\rpn.py in __init__(self, cfg, input_shape)
119 cfg, [input_shape[f] for f in self.in_features]
120 )
--> 121 self.box2box_transform = Box2BoxTransform(weights=cfg.MODEL.RPN.BBOX_REG_WEIGHTS)
122 self.anchor_matcher = Matcher(
123 cfg.MODEL.RPN.IOU_THRESHOLDS, cfg.MODEL.RPN.IOU_LABELS, allow_low_quality_matches=True

c:users\rgooch\documents\detectron2_repo\detectron2\modeling\box_regression.py in __init__(self, weights, scale_clamp)
34 """
35 self.weights = weights
---> 36 assert(len(weights)==4),"Expecting a 4-elements tuple (for scaling (dx,dy,dw,dh))"
37 self.scale_clamp = scale_clamp
38

AssertionError: Expecting a 4-elements tuple (for scaling (dx,dy,dw,dh))
```

I suspect that the RRPN class in rrpn.py is trying to initialize the RPN class, which has in its __init __ block a call to Box2BoxTransform, which expects 4 weights, and receives the 5 weights from RRPN.

What do you think?

[ppwwyyxx]

Putting out a full example is not on our todo list for near term, but to unblock some users, what you need to train a Rotated Faster R-CNN is the following changes to config:

MODEL:
  ANCHOR_GENERATOR:
    NAME: RotatedAnchorGenerator
    ANGLES: [[-90,-60,-30,0,30,60,90]]
  PROPOSAL_GENERATOR:
    NAME: RRPN
  RPN:
    BBOX_REG_WEIGHTS: (1,1,1,1,1)
  ROI_BOX_HEAD:
    POOLER_TYPE: ROIAlignRotated
    BBOX_REG_WEIGHTS: (10,10,5,5,1)
  ROI_HEADS:
    NAME: RROIHeads

plus a custom mapper in the data loader to handle rotated boxes. Some notable changes needed in the mapper:

• utils.transform_instance_annotations does not work for rotated boxes and you need a custom version using transform.apply_rotated_box.
• utils.annotations_to_instances needs to be replaced by utils.annotations_to_instances_rotated.

Plus a custom dataset that contains boxes with mode XYWHA.

[st7ma784]

Awesome! Many thanks, as for an example, I'm working on a dissertation in colab notebook environment, when finished I will happily share a cleaned up version here as an example.

Get Outlook for Androidhttps://aka.ms/ghei36

---

From: Yuxin Wu [e-mail]
Sent: Friday, March 6, 2020 12:58:27 AM
To: facebookresearch/detectron2 [e-mail]
Cc: Mander, Stephen (Student) [e-mail]; Mention [e-mail]
Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

Putting out a full example is not on our todo list for near term, but to unblock some users, what you need to train a Rotated Faster R-CNN is the following changes to config:

MODEL:
ANCHOR_GENERATOR:
NAME: RotatedAnchorGenerator
ANGLES: [[-90,-60,-30,0,30,60,90]]
PROPOSAL_GENERATOR:
NAME: RRPN
RPN:
BBOX_REG_WEIGHTS: (1,1,1,1,1)
ROI_BOX_HEAD:
POOLER_TYPE: ROIAlignRotated
BBOX_REG_WEIGHTS: (10,10,5,5,1)
ROI_HEADS:
NAME: RROIHeads

plus a custom mapper in the data loaderhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdetectron2.readthedocs.io%2Ftutorials%2Fdata_loading.html%23write-a-custom-dataloader&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C692e1018e5aa4ef0400808d7c1697e84%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637190531150368167&sdata=N3BRcKXz9QhK5CkGNQG74msjjMSf1xXKynhi3K2NpV8%3D&reserved=0 to handle rotated boxes. Some notable changes needed in the mapper:

• utils.transform_instance_annotations does not work for rotated boxes and you need a custom version using transform.apply_rotated_box.
• utils.annotations_to_instances needs to be replaced by utils.annotations_to_instances_rotated.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TU6MQNUAZHDMAM2H66DRGBDDHA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN7PGDQ%23issuecomment-595522318&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C692e1018e5aa4ef0400808d7c1697e84%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637190531150378165&sdata=6hTd03x22fkXHHU8lTETKVu1Q%2B%2B1etGCMw9%2Bw6NlKY0%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU3KD6LSQ7AKGJ3KY53RGBDDHANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C692e1018e5aa4ef0400808d7c1697e84%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637190531150378165&sdata=lle9RdHjc%2FjWrXJZ8bS%2BzpGdOHoaXWcsv0xyo9oS7TE%3D&reserved=0.

[st7ma784]

My training now works, cheers @ppwwyyxx!

for anyone else struggling, heres the cell that made a difference:

from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg
from detectron2.modeling import build_model
from detectron2.evaluation import inference_on_dataset, RotatedCOCOEvaluator
from detectron2 import model_zoo
from detectron2.data import transforms as T
from detectron2.data import detection_utils as utils
from detectron2.data import DatasetCatalog, MetadataCatalog, build_detection_test_loader,build_detection_train_loader
import copy
from detectron2.engine import DefaultTrainer, default_argument_parser, default_setup, hooks, launch

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml") # Let training initialize from model zoo
cfg.DATASETS.TRAIN = (['Train'])
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 4
cfg.MODEL.MASK_ON=False
cfg.SOLVER.IMS_PER_BATCH = 4
cfg.SOLVER.CHECKPOINT_PERIOD=2000
cfg.SOLVER.BASE_LR = 0.0005  # pick a good LR
cfg.SOLVER.MAX_ITER=10000   # 300 iterations seems good enough for this toy dataset; you may need to train longer for a practical dataset
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512   # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_HEADS.NUM_CLASSES =ClassCount  
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set threshold for this model
cfg.MODEL.PROPOSAL_GENERATOR.NAME = "RRPN"
cfg.MODEL.RPN.HEAD_NAME = "StandardRPNHead"
cfg.MODEL.RPN.BBOX_REG_WEIGHTS = (10,10,5,5,1)
cfg.MODEL.ANCHOR_GENERATOR.NAME = "RotatedAnchorGenerator"
cfg.MODEL.ANCHOR_GENERATOR.ANGLES = [[-90,-60,-30,0,30,60,90]]
cfg.MODEL.ROI_HEADS.NAME = "RROIHeads"
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256 # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_BOX_HEAD.POOLER_TYPE = "ROIAlignRotated"
cfg.MODEL.ROI_BOX_HEAD.BBOX_REG_WEIGHTS = (10.0, 10.0, 5.0, 5.0, 10.0)
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)#

def my_transform_instance_annotations(annotation, transforms, image_size, *, keypoint_hflip_indices=None):
  annotation["bbox"] = transforms.apply_rotated_box(np.asarray([annotation['bbox']]))[0]
  annotation["bbox_mode"] = BoxMode.XYXY_ABS
  return annotation

def mapper(dataset_dict):
  dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
  image = utils.read_image(dataset_dict["file_name"], format="BGR")
  image, transforms = T.apply_transform_gens([T.Resize((800, 800))], image)
  dataset_dict["image"] = torch.as_tensor(image.transpose(2, 0, 1).astype("float32"))
  annos = [
      my_transform_instance_annotations(obj, transforms, image.shape[:2])  
      for obj in dataset_dict.pop("annotations")
      if obj.get("iscrowd", 0) == 0
  ]
  instances = utils.annotations_to_instances_rotated(annos, image.shape[:2])
  dataset_dict["instances"] = utils.filter_empty_instances(instances)
  return dataset_dict

class MyTrainer(DefaultTrainer):
  @classmethod
  def build_evaluator(cls, cfg, dataset_name):
      output_folder = os.path.join(cfg.OUTPUT_DIR, "inference")
      evaluators = [RotatedCOCOEvaluator(dataset_name, cfg, True, output_folder)]
      return DatasetEvaluators(evaluators)
  @classmethod
  def build_train_loader(cls, cfg):
        return build_detection_train_loader(cfg,mapper=mapper)

EDIT Changed evaluator to ROTATEDcocoevaluator

[kakaluote]

My training is working，but it seem not convergence

[breznak]

@st7ma784 could you reformat your post with code (tripe `) to be more readable?
Also, would you like to commit the rotated dataloader for the PR #996 ?

[st7ma784]

@breznak sure- a little new to gtihub with not just my own project so not sure how. (this is one of my first "real world" projects using github)

[st7ma784]

@kakaluote I found the same, with a very low LR, (I was using 0.0005) after 6000 iters there was no convergence, upping it to 0.005 helped massively, but I suspect its to do with the RegBox weights having angle as least. I think that might depend on the object shapes in the dataset, if your object are long and thin (Im using Ships) then the angle is far more important than the X\Y of a BBOX proposal. try changing cfg.MODEL.RPN.BBOX_REG_WEIGHTS or cfg.MODEL.ROI_BOX_HEAD.BBOX_REG_WEIGHTS back to (1,1,1,1,1)

[Amir22010]

@st7ma784 you solved my training issue thanks for help:))))))))

i use your custom mapper function and modify as per your cfg params

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = MyTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

now i my training has started

[Amir22010]

@kakaluote I found the same, with a very low LR, (I was using 0.0005) after 6000 iters there was no convergence, upping it to 0.005 helped massively, but I suspect its to do with the RegBox weights having angle as least. I think that might depend on the object shapes in the dataset, if your object are long and thin (Im using Ships) then the angle is far more important than the X\Y of a BBOX proposal. try changing cfg.MODEL.RPN.BBOX_REG_WEIGHTS or cfg.MODEL.ROI_BOX_HEAD.BBOX_REG_WEIGHTS back to (1,1,1,1,1)

can i change both the param to (1,1,1,1,1)???

[st7ma784]

I presume so, still waiting for other tests in colab to run... also to check correct angles of BBOXes 👍

import random
dataset_dicts = returnTRAINRECORDS()
print(ClassesTotals)
class myVisualizer(Visualizer):

    def draw_dataset_dict(self, dic):
        annos = dic.get("annotations", None)
        if annos:
            if "segmentation" in annos[0]:
                masks = [x["segmentation"] for x in annos]
            else:
                masks = None
            if "keypoints" in annos[0]:
                keypts = [x["keypoints"] for x in annos]
                keypts = np.array(keypts).reshape(len(annos), -1, 3)
            else:
                keypts = None

            boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYWHA_ABS) for x in annos]

            labels = [x["category_id"] for x in annos]
            names = self.metadata.get("thing_classes", None)
            if names:
                labels = [names[i] for i in labels]
            labels = [
                "{}".format(i) + ("|crowd" if a.get("iscrowd", 0) else "")
                for i, a in zip(labels, annos)
            ]
            self.overlay_instances(labels=labels, boxes=boxes, masks=masks, keypoints=keypts)

        sem_seg = dic.get("sem_seg", None)
        if sem_seg is None and "sem_seg_file_name" in dic:
            sem_seg = cv2.imread(dic["sem_seg_file_name"], cv2.IMREAD_GRAYSCALE)
        if sem_seg is not None:
            self.draw_sem_seg(sem_seg, area_threshold=0, alpha=0.5)
        return self.output
for d in random.sample(dataset_dicts, 3):
    e=dict(d)
    name=e.get("file_name")
    print(name)
    print(e.items())
    img = cv2.imread(name)
    visualizer = myVisualizer(img[:, :, ::-1], metadata={}, scale=1)
    vis = visualizer.draw_dataset_dict(e, )
    cv2_imshow(vis.get_image()[:, :, ::-1])

[Amir22010]

I presume so, still waiting for other tests in colab to run... also to check correct angles of BBOXes 👍

import random
dataset_dicts = returnTRAINRECORDS()
print(ClassesTotals)
class myVisualizer(Visualizer):

    def draw_dataset_dict(self, dic):
        annos = dic.get("annotations", None)
        if annos:
            if "segmentation" in annos[0]:
                masks = [x["segmentation"] for x in annos]
            else:
                masks = None
            if "keypoints" in annos[0]:
                keypts = [x["keypoints"] for x in annos]
                keypts = np.array(keypts).reshape(len(annos), -1, 3)
            else:
                keypts = None

            boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYWHA_ABS) for x in annos]

            labels = [x["category_id"] for x in annos]
            names = self.metadata.get("thing_classes", None)
            if names:
                labels = [names[i] for i in labels]
            labels = [
                "{}".format(i) + ("|crowd" if a.get("iscrowd", 0) else "")
                for i, a in zip(labels, annos)
            ]
            self.overlay_instances(labels=labels, boxes=boxes, masks=masks, keypoints=keypts)

        sem_seg = dic.get("sem_seg", None)
        if sem_seg is None and "sem_seg_file_name" in dic:
            sem_seg = cv2.imread(dic["sem_seg_file_name"], cv2.IMREAD_GRAYSCALE)
        if sem_seg is not None:
            self.draw_sem_seg(sem_seg, area_threshold=0, alpha=0.5)
        return self.output
for d in random.sample(dataset_dicts, 3):
    e=dict(d)
    name=e.get("file_name")
    print(name)
    print(e.items())
    img = cv2.imread(name)
    visualizer = myVisualizer(img[:, :, ::-1], metadata={}, scale=1)
    vis = visualizer.draw_dataset_dict(e, )
    cv2_imshow(vis.get_image()[:, :, ::-1])

its not working still getting straight boxes,,notgetting angled/rotated boxes while doing prediction from saved model,,,,,,,,i think it will be annotation["bbox_mode"] = BoxMode.XYWHA_ABS in your my_transform_instance_annotations()

[st7ma784]

If you look at how boxmode converts, if you are using a presaved model, Im presuming the output of that is naturally ‘BoxMode.XYXY_ABS’ converting this to XYWHA doesn’t make it smaller it simply changes makes it XYWH and appends a 0 for the angle.

For reference, that code is used after I load in my dataset so I can verify its correct pre training.

---

From: Amir Khan [e-mail]
Sent: Friday, March 6, 2020 7:23:32 PM
To: facebookresearch/detectron2 [e-mail]
Cc: Mander, Stephen (Student) [e-mail]; Mention [e-mail]
Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

I presume so, still waiting for other tests in colab to run... also to check correct angles of BBOXes 👍

import random

dataset_dicts = returnTRAINRECORDS()

print(ClassesTotals)

class myVisualizer(Visualizer):

def draw_dataset_dict(self, dic):

    annos = dic.get("annotations", None)

    if annos:

        if "segmentation" in annos[0]:

            masks = [x["segmentation"] for x in annos]

        else:

            masks = None

        if "keypoints" in annos[0]:

            keypts = [x["keypoints"] for x in annos]

            keypts = np.array(keypts).reshape(len(annos), -1, 3)

        else:

            keypts = None



        boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYWHA_ABS) for x in annos]



        labels = [x["category_id"] for x in annos]

        names = self.metadata.get("thing_classes", None)

        if names:

            labels = [names[i] for i in labels]

        labels = [

            "{}".format(i) + ("|crowd" if a.get("iscrowd", 0) else "")

            for i, a in zip(labels, annos)

        ]

        self.overlay_instances(labels=labels, boxes=boxes, masks=masks, keypoints=keypts)



    sem_seg = dic.get("sem_seg", None)

    if sem_seg is None and "sem_seg_file_name" in dic:

        sem_seg = cv2.imread(dic["sem_seg_file_name"], cv2.IMREAD_GRAYSCALE)

    if sem_seg is not None:

        self.draw_sem_seg(sem_seg, area_threshold=0, alpha=0.5)

    return self.output

for d in random.sample(dataset_dicts, 3):

e=dict(d)

name=e.get("file_name")

print(name)

print(e.items())

img = cv2.imread(name)

visualizer = myVisualizer(img[:, :, ::-1], metadata={}, scale=1)

vis = visualizer.draw_dataset_dict(e, )

cv2_imshow(vis.get_image()[:, :, ::-1])

its not working still getting straight boxes,,notgetting angled/rotated boxes while doing prediction from saved model

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TU2XZ2YEK6CDYD6VHXLRGFETJA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCQXZA%23issuecomment-595921892&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171292922&sdata=eYVBUB5tIbczFpvGE9e5GbH9RoE%2FNQZd8xVpZLQjSIc%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU4NHXMDFA7MRSQWWTTRGFETJANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171302916&sdata=WYrS9e02EXMV0g12gyf%2FIkuAuf5hPgrkXjyN9xB4uBE%3D&reserved=0.

[Amir22010]

My training now works, cheers @ppwwyyxx!

for anyone else struggling, heres the cell that made a difference:

from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg
from detectron2.modeling import build_model
from detectron2.evaluation import inference_on_dataset, COCOEvaluator
from detectron2 import model_zoo
from detectron2.data import transforms as T
from detectron2.data import detection_utils as utils
from detectron2.data import DatasetCatalog, MetadataCatalog, build_detection_test_loader,build_detection_train_loader
import copy
from detectron2.engine import DefaultTrainer, default_argument_parser, default_setup, hooks, launch

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml") # Let training initialize from model zoo
cfg.DATASETS.TRAIN = (['Train'])
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 4
cfg.MODEL.MASK_ON=False
cfg.SOLVER.IMS_PER_BATCH = 4
cfg.SOLVER.CHECKPOINT_PERIOD=2000
cfg.SOLVER.BASE_LR = 0.0005  # pick a good LR
cfg.SOLVER.MAX_ITER=10000   # 300 iterations seems good enough for this toy dataset; you may need to train longer for a practical dataset
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512   # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_HEADS.NUM_CLASSES =ClassCount  
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set threshold for this model
cfg.MODEL.PROPOSAL_GENERATOR.NAME = "RRPN"
cfg.MODEL.RPN.HEAD_NAME = "StandardRPNHead"
cfg.MODEL.RPN.BBOX_REG_WEIGHTS = (10,10,5,5,1)
cfg.MODEL.ANCHOR_GENERATOR.NAME = "RotatedAnchorGenerator"
cfg.MODEL.ANCHOR_GENERATOR.ANGLES = [[-90,-60,-30,0,30,60,90]]
cfg.MODEL.ROI_HEADS.NAME = "RROIHeads"
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256 # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_BOX_HEAD.POOLER_TYPE = "ROIAlignRotated"
cfg.MODEL.ROI_BOX_HEAD.BBOX_REG_WEIGHTS = (10.0, 10.0, 5.0, 5.0, 10.0)
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)#

def my_transform_instance_annotations(annotation, transforms, image_size, *, keypoint_hflip_indices=None):
  annotation["bbox"] = transforms.apply_rotated_box(np.asarray([annotation['bbox']]))[0]
  annotation["bbox_mode"] = BoxMode.XYXY_ABS
  return annotation

def mapper(dataset_dict):
  dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
  image = utils.read_image(dataset_dict["file_name"], format="BGR")
  image, transforms = T.apply_transform_gens([T.Resize((800, 800))], image)
  dataset_dict["image"] = torch.as_tensor(image.transpose(2, 0, 1).astype("float32"))
  annos = [
      my_transform_instance_annotations(obj, transforms, image.shape[:2])  
      for obj in dataset_dict.pop("annotations")
      if obj.get("iscrowd", 0) == 0
  ]
  instances = utils.annotations_to_instances_rotated(annos, image.shape[:2])
  dataset_dict["instances"] = utils.filter_empty_instances(instances)
  return dataset_dict

class MyTrainer(DefaultTrainer):
  @classmethod
  def build_evaluator(cls, cfg, dataset_name):
      output_folder = os.path.join(cfg.OUTPUT_DIR, "inference")
      evaluators = [COCOEvaluator(dataset_name, cfg, True, output_folder)]
      return DatasetEvaluators(evaluators)
  @classmethod
  def build_train_loader(cls, cfg):
        return build_detection_train_loader(cfg,mapper=mapper)

,i think it will be annotation["bbox_mode"] = BoxMode.XYWHA_ABS in your my_transform_instance_annotations()

[Amir22010]

If you look at how boxmode converts, if you are using a presaved model, Im presuming the output of that is naturally ‘BoxMode.XYXY_ABS’ converting this to XYWHA doesn’t make it smaller it simply changes makes it XYWH and appends a 0 for the angle. For reference, that code is used after I load in my dataset so I can verify its correct pre training.
…
________________________________ From: Amir Khan [e-mail] Sent: Friday, March 6, 2020 7:23:32 PM To: facebookresearch/detectron2 [e-mail] Cc: Mander, Stephen (Student) [e-mail]; Mention [e-mail] Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21) This email originated outside the University. Check before clicking links or attachments. I presume so, still waiting for other tests in colab to run... also to check correct angles of BBOXes 👍 import random dataset_dicts = returnTRAINRECORDS() print(ClassesTotals) class myVisualizer(Visualizer): def draw_dataset_dict(self, dic): annos = dic.get("annotations", None) if annos: if "segmentation" in annos[0]: masks = [x["segmentation"] for x in annos] else: masks = None if "keypoints" in annos[0]: keypts = [x["keypoints"] for x in annos] keypts = np.array(keypts).reshape(len(annos), -1, 3) else: keypts = None boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYWHA_ABS) for x in annos] labels = [x["category_id"] for x in annos] names = self.metadata.get("thing_classes", None) if names: labels = [names[i] for i in labels] labels = [ "{}".format(i) + ("|crowd" if a.get("iscrowd", 0) else "") for i, a in zip(labels, annos) ] self.overlay_instances(labels=labels, boxes=boxes, masks=masks, keypoints=keypts) sem_seg = dic.get("sem_seg", None) if sem_seg is None and "sem_seg_file_name" in dic: sem_seg = cv2.imread(dic["sem_seg_file_name"], cv2.IMREAD_GRAYSCALE) if sem_seg is not None: self.draw_sem_seg(sem_seg, area_threshold=0, alpha=0.5) return self.output for d in random.sample(dataset_dicts, 3): e=dict(d) name=e.get("file_name") print(name) print(e.items()) img = cv2.imread(name) visualizer = myVisualizer(img[:, :, ::-1], metadata={}, scale=1) vis = visualizer.draw_dataset_dict(e, ) cv2_imshow(vis.get_image()[:, :, ::-1]) its not working still getting straight boxes,,notgetting angled/rotated boxes while doing prediction from saved model — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TU2XZ2YEK6CDYD6VHXLRGFETJA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCQXZA%23issuecomment-595921892&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171292922&sdata=eYVBUB5tIbczFpvGE9e5GbH9RoE%2FNQZd8xVpZLQjSIc%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU4NHXMDFA7MRSQWWTTRGFETJANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171302916&sdata=WYrS9e02EXMV0g12gyf%2FIkuAuf5hPgrkXjyN9xB4uBE%3D&reserved=0.

so, do u know which pre saved model has boxmode XYWHA_ABS as output???,,,or i have to train it from scratch

[st7ma784]

Train from scratch I’m afraid. When a checkpoint file is loaded in there’s often a

‘’’
'proposal_generator.anchor_generator.cell_anchors.0' has shape (3, 4) in the checkpoint but (21, 5) in the model! Skipped.
'proposal_generator.anchor_generator.cell_anchors.1' has shape (3, 4) in the checkpoint but (21, 5) in the model! Skipped.
'proposal_generator.anchor_generator.cell_anchors.2' has shape (3, 4) in the checkpoint but (21, 5) in the model! Skipped.
'proposal_generator.anchor_generator.cell_anchors.3' has shape (3, 4) in the checkpoint but (21, 5) in the model! Skipped.
'proposal_generator.anchor_generator.cell_anchors.4' has shape (3, 4) in the checkpoint but (21, 5) in the model! Skipped.
‘’’
Meaning the model is using a different set of layer shapes then the pretrained bit.

Coincidently, this is why colab is such a great environment, the access to decent GPUs for short stints of training with transferred learning.

---

From: Amir Khan [e-mail]
Sent: Friday, March 6, 2020 7:46:03 PM
To: facebookresearch/detectron2 [e-mail]
Cc: Mander, Stephen (Student) [e-mail]; Mention [e-mail]
Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

If you look at how boxmode converts, if you are using a presaved model, Im presuming the output of that is naturally ‘BoxMode.XYXY_ABS’ converting this to XYWHA doesn’t make it smaller it simply changes makes it XYWH and appends a 0 for the angle. For reference, that code is used after I load in my dataset so I can verify its correct pre training.
…
________________________________ From: Amir Khan [email protected][e-mail] Sent: Friday, March 6, 2020 7:23:32 PM To: facebookresearch/detectron2 [email protected][e-mail] Cc: Mander, Stephen (Student) [email protected][e-mail]; Mention [email protected][e-mail] Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6f248ce8dbef498ed6d908d7c20703ea%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191207806448410&sdata=pvk6aJxWyApuq%2BFz2IE5XhKrY5mxq21cThCn9wBFYug%3D&reserved=0) This email originated outside the University. Check before clicking links or attachments. I presume so, still waiting for other tests in colab to run... also to check correct angles of BBOXes 👍 import random dataset_dicts = returnTRAINRECORDS() print(ClassesTotals) class myVisualizer(Visualizer): def draw_dataset_dict(self, dic): annos = dic.get("annotations", None) if annos: if "segmentation" in annos[0]: masks = [x["segmentation"] for x in annos] else: masks = None if "keypoints" in annos[0]: keypts = [x["keypoints"] for x in annos] keypts = np.array(keypts).reshape(len(annos), -1, 3) else: keypts = None boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYWHA_ABS) for x in annos] labels = [x["category_id"] for x in annos] names = self.metadata.get("thing_classes", None) if names: labels = [names[i] for i in labels] labels = [ "{}".format(i) + ("|crowd" if a.get("iscrowd", 0) else "") for i, a in zip(labels, annos) ] self.overlay_instances(labels=labels, boxes=boxes, masks=masks, keypoints=keypts) sem_seg = dic.get("sem_seg", None) if sem_seg is None and "sem_seg_file_name" in dic: sem_seg = cv2.imread(dic["sem_seg_file_name"], cv2.IMREAD_GRAYSCALE) if sem_seg is not None: self.draw_sem_seg(sem_seg, area_threshold=0, alpha=0.5) return self.output for d in random.sample(dataset_dicts, 3): e=dict(d) name=e.get("file_name") print(name) print(e.items()) img = cv2.imread(name) visualizer = myVisualizer(img[:, :, ::-1], metadata={}, scale=1) vis = visualizer.draw_dataset_dict(e, ) cv2_imshow(vis.get_image()[:, :, ::-1]) its not working still getting straight boxes,,notgetting angled/rotated boxes while doing prediction from saved model — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TU2XZ2YEK6CDYD6VHXLRGFETJA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCQXZA%23issuecomment-595921892&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171292922&sdata=eYVBUB5tIbczFpvGE9e5GbH9RoE%2FNQZd8xVpZLQjSIc%3D&reserved=0https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TU2XZ2YEK6CDYD6VHXLRGFETJA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCQXZA%23issuecomment-595921892&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6f248ce8dbef498ed6d908d7c20703ea%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191207806448410&sdata=UThDr9DIcEaMXhRW%2B7xHODMXGLQXg2VsYExY7eHN%2BRY%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU4NHXMDFA7MRSQWWTTRGFETJANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7Cf7fbc413f4204d69d90708d7c203ddfb%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191194171302916&sdata=WYrS9e02EXMV0g12gyf%2FIkuAuf5hPgrkXjyN9xB4uBE%3D&reserved=0https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU4NHXMDFA7MRSQWWTTRGFETJANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6f248ce8dbef498ed6d908d7c20703ea%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191207806458408&sdata=7%2FJd%2FxN1uFZmvy1OodmR3U2JStzr398lY7WiToF8brU%3D&reserved=0.

so, do u know which pre saved model has boxmode XYWHA_ABS as output???,,,or i have to train it from scratch

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TUYUTJYIBI3G53PHVUTRGFHHXA5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOCS7DI%23issuecomment-595931021&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6f248ce8dbef498ed6d908d7c20703ea%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191207806458408&sdata=2dwl2wOJeAf29XFS8pH009o3JL%2FFZ1yoQx5JnySup9c%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TUZUEZ55AJYFHVS6COLRGFHHXANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6f248ce8dbef498ed6d908d7c20703ea%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637191207806468404&sdata=xWu%2FdUVqz%2F5qJCl45HwWsWOaLVG1KwL4Osh5DHir8eo%3D&reserved=0.

[kakaluote]

@st7ma784 Could you share the RotatedCOCOEvaluator？

[st7ma784]

I think the code you’re after is:
‘’’
from detectron2.evaluation import RotatedCOCOEvaluator
‘’’

Hope that helps

---

From: kakaluote [e-mail]
Sent: Monday, March 9, 2020 7:51:27 AM
To: facebookresearch/detectron2 [e-mail]
Cc: Mander, Stephen (Student) [e-mail]; Mention [e-mail]
Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

@st7ma784https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fst7ma784&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6ac610bbd5804f7257dd08d7c3feaee0%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637193370942416625&sdata=QPFYZ7A7cDEHIemGr%2F0ojpVp87mJ7dhkxnoOCEm148c%3D&reserved=0 Could you share the RotatedCOCOEvaluator？

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%3Femail_source%3Dnotifications%26email_token%3DAI25TUZQEXLUT2ZOI7UOOHDRGSNX7A5CNFSM4JABAQJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOGAJRI%23issuecomment-596378821&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6ac610bbd5804f7257dd08d7c3feaee0%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637193370942426619&sdata=C2uPc%2BWrg54oenV7hwqmQAfuetuUHtWUASIotoJ5V2M%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU6TPYUZBZQ5BTQV6DLRGSNX7ANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C6ac610bbd5804f7257dd08d7c3feaee0%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637193370942426619&sdata=dsWVgkLnO41kBiVNrywUsilPRF3g2%2BUA%2Bk5suOGdnJg%3D&reserved=0.

[kakaluote]

thank you, the new version has RotatedCOCOEvaluator

[cbasavaraj]

Hi @st7ma784 ,
I don't think fvcore has transforms.apply_rotated_box(). Did you define this yourself?
Thanks

[st7ma784]

Nope, its in the detectron2 docs.

Im still trying to get a good eval for Rotated bounding boxes, I think it comes down to RotatedCocoEvaluator still being reliant on the coco api calling maskutils method for IOU.

The lazy way is to simply convert box back to a non rotated bounding box but this can yield false positives, the better way is to rewrite the method ☹

Thats tonights problem.
From: Chandrachud [e-mail]
Sent: 13 March 2020 17:35
To: facebookresearch/[e-mail]
Cc: Mander, Stephen (Student)[e-mail]; [e-mail]
Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

Hi @st7ma784https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fst7ma784&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C58713d258d304804df2708d7c774eac0%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637197177271847578&sdata=wrobrhZORRR1yURy%2BEbBrs7QkPHYtzzicm3nB6dXG8o%3D&reserved=0 ,
I don't think fvcore has transforms.apply_rotated_box(). Did you define this yourself?
Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%23issuecomment-598835991&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C58713d258d304804df2708d7c774eac0%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637197177271857578&sdata=0%2BkIMYkButhvknLTFUGpg0k5w3WllIyuJ%2BEXzsb4vgA%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU73I46MIRNOPWN5KVLRHJVFXANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C58713d258d304804df2708d7c774eac0%7C9c9bcd11977a4e9ca9a0bc734090164a%7C1%7C0%7C637197177271857578&sdata=6uDzriVwYv3QpU7caseDtZmP74J%2FrUrIeuyLgA2z2hg%3D&reserved=0.

[NO2-yh]

Any one now works well with rotated faster rcnn? Waiting for some updates.......

[st7ma784]

Yep, mine does, apologies as I forgot to add an update, owing to having to actually write my dissertation. (turns out just telling your supervisor you understand isn’t enough 😝)

I found that training takes a fair bit longer so haven’t had any good outcomes from Colab yet. Best results came from a Mask-rcnn network with rotated boxes. (mask rcnn as I understand it being a further evolution of faster-rcnn)

From: [e-mail]
Sent: 02 April 2020 16:59
To: facebookresearch/[e-mail]
Cc: Mander, Stephen (Student)[e-mail]; [e-mail]
Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

Any one now works well with rotated faster rcnn? Waiting for some updates.......

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%23issuecomment-607934930&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C195ae2b78c684afd589908d7d71ee08d%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637214399924707607&sdata=wYO%2BZyNkXXBk8n4S7uiTrzBn9Fc9ctuNtXHQHS44hhQ%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU66ZNNOVNHCYSJH7MDRKSY7JANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C195ae2b78c684afd589908d7d71ee08d%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637214399924707607&sdata=o8Nzjum2SdyvjnJGa0wH6F6fLDWb3uivk9eaEUf5o%2F0%3D&reserved=0.

[NO2-yh]

@st7ma784 Thanks for your update :)
So, which annotation format should i prepare my custom dataset ? bbox: x1, y1, x2, y2 or box: xc, yc, w, h, angle

[st7ma784]

The latter: xc,yx,w,h,a

---

From: yh673025667 [e-mail]
Sent: Thursday, April 2, 2020 5:28:30 PM
To: facebookresearch/detectron2 [e-mail]
Cc: Mander, Stephen (Student) [e-mail]; Mention [e-mail]
Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

@st7ma784https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fst7ma784&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C94989bcb6381412bb61208d7d722e319%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637214417143194270&sdata=FpwS%2Fd0HPIMMVwUZNX27LV%2FuK%2FomTrbQslrscF8Z09g%3D&reserved=0 Thanks for your update :)
So, which annotation format should i prepare my custom dataset ? bbox: x1, y1, x2, y2 or box: xc, yc, w, h, angle

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%23issuecomment-607951599&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C94989bcb6381412bb61208d7d722e319%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637214417143204260&sdata=TgzvVhUNZQBuGerPSEc6%2BbfX3Uzu2NLQOECuS7CVesM%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU5X72GRRYAMUM4C35TRKS4K5ANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C94989bcb6381412bb61208d7d722e319%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637214417143204260&sdata=itzAMPI%2BRIfn1V%2FyUB%2BUaEo2T%2F07pcCieO3v5nQ3Lw4%3D&reserved=0.

[NO2-yh]

@st7ma784 Can you share your rotated maskrcnn config settings? much appreciate. ^_^

[NO2-yh]

Traceback (most recent call last): File "tools/rotated_train_net.py", line 109, in <module> args=(args,), File "/home/work/project/detectron2/detectron2/engine/launch.py", line 52, in launch main_func(*args) File "tools/rotated_train_net.py", line 91, in main trainer = MyTrainer(cfg) File "/home/work/project/detectron2/detectron2/engine/defaults.py", line 253, in __init__ model = self.build_model(cfg) File "/home/work/project/detectron2/detectron2/engine/defaults.py", line 397, in build_model model = build_model(cfg) File "/home/work/project/detectron2/detectron2/modeling/meta_arch/build.py", line 19, in build_model return META_ARCH_REGISTRY.get(meta_arch)(cfg) File "/home/work/project/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 35, in __init__ self.roi_heads = build_roi_heads(cfg, self.backbone.output_shape()) File "/home/work/project/detectron2/detectron2/modeling/roi_heads/roi_heads.py", line 42, in build_roi_heads return ROI_HEADS_REGISTRY.get(name)(cfg, input_shape) File "/home/work/project/detectron2/detectron2/modeling/roi_heads/rotated_fast_rcnn.py", line 172, in __init__ ), "Mask/Keypoints not supported in Rotated ROIHeads." AssertionError: Mask/Keypoints not supported in Rotated ROIHeads.

Seems like haven't support rotated mask rcnn yet, only just support faster rcnn

[jackkwok]

@ppwwyyxx : can you give some guidance on adding support for mask head to rotated_fast_rcnn.py? Your hints above got me to the same point as @yh673025667.

[NO2-yh]

Waiting for some updates.......

[spytensor]

Waiting for some updates too .......

[ckybit]

Hi @st7ma784 ,
I don't think fvcore has transforms.apply_rotated_box(). Did you define this yourself?
Thanks

i can't find transforms.apply_rotated_box() in detectron2 and fvcore too

[st7ma784]

I’ll post a link to my colab notebook on Thursday, I’ve been a little wary of sharing code as its for my dissertation. The import I used is

‘’’from detectron2.data import transforms’’’

From: [e-mail]
Sent: 25 May 2020 14:48
To: facebookresearch/[e-mail]
Cc: Mander, Stephen (Student)[e-mail]; [e-mail]
Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

Hi @st7ma784https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fst7ma784&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C32cdf5353cea4b2d0e1d08d800b24979%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637260113019025124&sdata=ajhxkL5Xc6GhUVLkP7IkEuwN9O9RUqyjSXtEfW6JwOU%3D&reserved=0 ,
I don't think fvcore has transforms.apply_rotated_box(). Did you define this yourself?
Thanks

i can't find transforms.apply_rotated_box() in detectron2 and fvcore too

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%23issuecomment-633581593&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C32cdf5353cea4b2d0e1d08d800b24979%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637260113019025124&sdata=I45JdsKoiHg5xU99t%2FnwFad%2FTIOe%2B3b3%2BamRC09jNss%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU7ES3WBA3ODUAJ2MR3RTJZKDANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C32cdf5353cea4b2d0e1d08d800b24979%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637260113019035119&sdata=CSUUmIHHVbjJiDjrer5BescFPQebCc6R5PPyOz%2BXZX0%3D&reserved=0.

[st7ma784]

Apologies for delay:

Here's a stripped back notebook from my dissertation - Its pulling in data I've stored on my Google Drive so you'll need to rewrite the dataloader before trying it.

https://colab.research.google.com/drive/1ijCf9LwJ3HbXMBFXi4CM6qd5IBXlh330?usp=sharing

[NO2-yh]

@st7ma784 Hi, you set the MASK_ON to False, that means your work is still based rotated faster rcnn but not rotated mask rcnn

[st7ma784]

Ah, squints at issue title

Have I indeed :-P ?

[adamadatasci]

@st7ma784 Why are you using the pre-trained weight cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml")

And use annotation["bbox_mode"] = BoxMode.XYWHA_ABS ?

Where you able to accurately predict the ship orientation and localisation?

[st7ma784]

As a quirk of Colab, you aren't garunteed which GPU you get, and are limited to 12hrs. As such using pretrained weights massively speeds up training, (quite a few papers written about first chunk of training being largely about segmentation over RPN). From the dataset I had, XYWHA was the best option. but also, Its the only one with Angular data, thus I suspect the only one which works with the RRPN. (you can convert back to XYXY easily but the Boxmode convert isnt able to convert the other way)
And yes, pretty accurately. I was achieving about a 50% AP, which for a <1000 dataset spread unevenly across 20 classes is pretty good

[adamadatasci]

@st7ma784 Thanks for your work.

1 - You can use colab pro for 10 usd a month. Higher GPU and more than 24H without disconnection.

2 - I wasn't getting the fact your pre-trained model is in XYXY while your training data is in XYWHA.

3 - Are X,Y the center of the box in XYWHA ?

4 - I decided to train from scratch.

5 - It keeps crashing sometimes even with the pre-trained weight.

6 - I got a few trainings without crashing, but the boxes are oversized and outside the image.

7 - I'm using opencv to convert the inference back to normal bbox:

d =get_data()[500]
im_path = d["file_name"]
im = cv2.imread(im_path)

outputs = predictor(im)
bbox = outputs["instances"].pred_boxes.to("cpu").tensor.numpy().astype(float)

[cx,cy,w,h,a ] = bbox[0]
rect = ((cx, cy), (w, h), a)

box = cv2.boxPoints(rect)
box = np.int0(box)

lx,ly = box.T

x1,x2 = lx.min(),lx.max()
y1,y2 = ly.min(),ly.max()

#

Training Config

#

cfg = get_cfg()

cfg.merge_from_file(model_zoo.get_config_file("Base-RCNN-FPN.yaml"))

cfg.DATASETS.TRAIN = ([dataset_name])
cfg.DATASETS.TEST = ()

cfg.MODEL.MASK_ON=False
cfg.MODEL.KEYPOINTS_ON=False

mr,mg,mb = 243.59909935767192 ,243.60688569955659, 243.61098590081198
cfg.MODEL.PIXEL_MEAN = [mr,mg,mb]
cfg.MODEL.PIXEL_STD = [1.0, 1.0, 1.0]

cfg.MODEL.RESNETS.STRIDE_IN_1X1= False
cfg.MODEL.RESNETS.NUM_GROUPS= 32
cfg.MODEL.RESNETS.WIDTH_PER_GROUP= 8
cfg.MODEL.RESNETS.DEPTH = 101

cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.CHECKPOINT_PERIOD=2000
cfg.SOLVER.BASE_LR = 0.0025 # pick a good LR
cfg.SOLVER.MAX_ITER= 5000 #66000
cfg.SOLVER.GAMMA = 0.1

The iteration number to decrease learning rate by GAMMA.

cfg.SOLVER.STEPS= (1000, 4000)

cfg.MODEL.ROI_HEADS.NUM_CLASSES =6
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set threshold for this model

cfg.DATALOADER.NUM_WORKERS = 2 # Number of data loading threads
cfg.DATALOADER.SAMPLER_TRAIN= "RepeatFactorTrainingSampler" # Options: TrainingSampler, RepeatFactorTrainingSampler
cfg.DATALOADER.REPEAT_THRESHOLD=0.01 # Repeat threshold for RepeatFactorTrainingSampler
cfg.DATALOADER.FILTER_EMPTY_ANNOTATIONS = True # if True, the dataloader will filter out images that have no associated # annotations at train time.

cfg.MODEL.PROPOSAL_GENERATOR.NAME = "RRPN"

cfg.MODEL.RPN.HEAD_NAME = "StandardRPNHead"
cfg.MODEL.RPN.BBOX_REG_WEIGHTS = (1.0, 1.0, 1.0, 1.0,1.0) #10,10,5,5,1 #10,10,5,5,10

cfg.MODEL.RPN.BOUNDARY_THRESH = 5

cfg.MODEL.RPN.IOU_THRESHOLDS = [0.5, 0.9]
cfg.MODEL.RPN.BATCH_SIZE_PER_IMAGE = 256

cfg.MODEL.ANCHOR_GENERATOR.NAME = "RotatedAnchorGenerator"
cfg.MODEL.ANCHOR_GENERATOR.ANGLES = [[-90,-60, -30, 0, 30,60, 90]]

cfg.MODEL.ROI_HEADS.NAME = "RROIHeads"
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256

cfg.MODEL.ROI_BOX_HEAD.POOLER_TYPE = "ROIAlignRotated"
cfg.MODEL.ROI_BOX_HEAD.BBOX_REG_WEIGHTS = (1.0,1.0,1.0,1.0,1.0)

[st7ma784]

not-so-fun fact: Colab Pro isnt available in the UK :-( (would have made stuff a lot easier)
The pretrained model speeds up the learning. the the Feature extraction part of a Faster-RCNN network, the BBOX format is somewhat irrelevant, as this is then fed into a classification and region proposal layers. For the task I was looking at, some papers suggest a modified RRPN with feature extractions, as the base objects are relatively easy to identify within and image, but the classification and feature extraction is difficult.
I did have a lot of finicking with getting the XYWHA read in correctly: the XY are the center. It's why in my notebook I have a visualizer before the training as it helps ensure annotations are accurate.

[adamadatasci]

@st7ma784 Yes, your visualizer works perfectly. I'm looking into using it for inference too. Will update more on this. My boxes are lines of 1-pixel width with different orientations.

[st7ma784]

sounds fun... If you've got a dataset like that, perhaps tools designed for edge-detection might work well?
I've spent the last day or so grappling with Kaggle so I can get a demo dataset read in straight to the Colab environment so I can share (eventually) a fully working notebook
.

[adamadatasci]

@st7ma784 It's lines classification and detection, with certain lines like dimension lines should not be included. While others should be detected and classified. The perfect model for me would be rotated bbox with keypoints. The Ship dataset I thought is an open dataset: https://www.kaggle.com/c/airbus-ship-detection/data?select=train_v2
It would be interesting to compete on the Airbus Ship Detection Challenge using Detectron2 :)

[st7ma784]

It very much would be. Though I've been using the far smaller HRSC2016 Dataset, as my dissertation was less about exact detection and segmentation, rather than identifying the ships present. As such, I've been trying to investigate ways of accurately identifying ships if a training set only has a handful of samples. I.e submarines/smuggling vessels...

[SujoyDU]

@st7ma784 thank you very much for the colab code. Btw, for the evaluation part how did you generate the coco-format json file? because
convert_to_coco_json("Train",''.join("./output/"+"Train"+"_coco_format.json"))
this tries to convert _BoxMode = XYWHA_ABS_ to _BoxMode = XYWH_ABS_ in the convert_to_coco_dict(dataset_name) method in _coco.py_ file. It says that, the conversion is not supported. Could you give an idea how you acheived that?

[st7ma784]

Have a look behind the form in my latest version :

https://drive.google.com/file/d/1zjXXcyvh4nyBQXmp30U4rnJ3OwCSSgr1/view?usp=sharing

From: Sujoy [e-mail]
Sent: 04 July 2020 00:45
To: facebookresearch/[e-mail]
Cc: Mander, Stephen (Student)[e-mail]; [e-mail]
Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

@st7ma784https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fst7ma784&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C55d8b9c5fc844041a08f08d81fab2b10%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637294167310635910&sdata=%2FiExbMReb%2B0Q5H%2BRQ7EU%2BFX27dm2gFhfav3K7KTbtMg%3D&reserved=0 thank you very much for the colab code. Btw, for the evaluation part how did you generate the coco-format json file? because
convert_to_coco_json("Train",''.join("./output/"+"Train"+"_coco_format.json"))
this tries to convert BoxMode = XYWHA_ABS to BoxMode = XYWH_ABS in the convert_to_coco_dict(dataset_name) method in coco.py file. It says that, the conversion is not supported. Could you give an idea how you acheived that?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%23issuecomment-653696673&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C55d8b9c5fc844041a08f08d81fab2b10%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637294167310645909&sdata=u90jgKxj%2BdmRqiOIq4VZlESlK7yoKDSVWYZKds67wPM%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU55XAXMJO6UUP35HWTRZZURLANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C55d8b9c5fc844041a08f08d81fab2b10%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637294167310645909&sdata=EgKzaG06tsQFJ3xBdJGNj0%2By8NsZ9%2BI9ql9Ez9KyakI%3D&reserved=0.

[APeiZou]

ok I got rotated bounding box working on DOTA dataset:
https://captain-whu.github.io/DOTA/dataset.html

for training, after several epochs, it'll fail but I'm pretty sure the issue is with respect to invalid ground truth bounding boxes in my dataset.

@ppwwyyxx do you suggest I first train rrpn and then rroi heads rather than end to end?

For inference, get_deltas() calls torch.div_() which resulted in a pytorch error. I got past this by changing the in place version to torch.div(). Not sure if this is considered a bug.

I wrote a custom dataloader and data augmentation using imgaug.
Dataloader prepares the data in detectron2 input format (list of dictionaries)

As far as this issue goes, here are my configs that overwrite the base rcnn fpn config:

_BASE_: "./Base-RCNN-FPN.yaml"
MODEL:
PIXEL_MEAN: [103.530, 116.280, 123.675] # Pixel mean of imagenet dataset R/G/B?
WEIGHTS: "./model_final_a3ec72.pkl"
MASK_ON: False
RESNETS:
DEPTH: 101
PROPOSAL_GENERATOR:
NAME: "RRPN"
RPN:
HEAD_NAME: "StandardRPNHead"
BBOX_REG_WEIGHTS: (1.0, 1.0, 1.0, 1.0, 1.0)
ANCHOR_GENERATOR:
NAME: "RotatedAnchorGenerator"
ANGLES: [[0, 30, 60, 90]]
ASPECT_RATIOS: [[0.5, 1.0, 2.0]]
ROI_HEADS:
NAME: "RROIHeads"
NUM_CLASSES: 15 # number of foreground classes
ROI_BOX_HEAD:
POOLER_TYPE: "ROIAlignRotated"
BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0, 10.0)
SOLVER:
STEPS: (210000, 250000)
MAX_ITER: 10 # number of epochs ( I modified the plain net train from iterations to epochs)
CHECKPOINT_PERIOD: 1 # number of epochs
why My Evaluation always like this when I use the Retinanet with Rotated anchor
| AP | AP50 | AP75 | APs | APm | APl |
|:------:|:----- -:|:----- -:|:----- :|:--- --:|:------:|
| 0 | 0 | 0 | nan | nan | 0 |

[APeiZou]

imgaug

how to write the rotated box augmentation when image is doing augmentation @edwardchaos

[karan-shr]

Hi @st7ma784,
First of all, thanks for sharing your work. I was following your old colab notebook and also checked out the newer one. But as @SujoyDU had mentioned the convert_to_coco_json("Train",''.join("./output/"+"Train"+"_coco_format.json")) bit when trying to convert to coco-json throws this error NotImplementedError: Conversion from BoxMode 4 to 1 is not supported yet which is occuring cause we are trying to convert BoxMode.XYWHA_ABS to BoxMode.XWWH_ABS. Did you also get the similar error? I am using the same dataset as you. @SujoyDU did you find a workaround? Any help would be appreciated. Thanks guys.

[st7ma784]

You are quite right, I recall that error occurring. I suspect something changed behind the scenes... My notebook still runs fine after a minor modification to correct that conversion. (you may need to have a nosey behind the form to find the offending function)

https://drive.google.com/file/d/1zjXXcyvh4nyBQXmp30U4rnJ3OwCSSgr1/view?usp=sharing

From: Karan [e-mail]
Sent: 23 July 2020 00:28
To: facebookresearch/[e-mail]
Cc: Mander, Stephen (Student)[e-mail]; [e-mail]
Subject: [External] Re: [facebookresearch/detectron2] Example for rotated faster rcnn (#21)

This email originated outside the University. Check before clicking links or attachments.

Hi @st7ma784https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fst7ma784&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C09fa8d83f5c94b7bf62908d82e96f95a%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637310573251226452&sdata=eONIAlq%2FR8CHitAPwGtUZdarHBuTVCdUa41u7DHAm%2FU%3D&reserved=0,
First of all, thanks for sharing your work. I was following your old colab notebook and also checked out the newer one. But as @SujoyDUhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSujoyDU&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C09fa8d83f5c94b7bf62908d82e96f95a%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637310573251226452&sdata=Gkc3xUi3H7%2F%2BwIDmlOGPP5Rf0Zw8tZuUVwP8jjQY8go%3D&reserved=0 had mentioned the convert_to_coco_json("Train",''.join("./output/"+"Train"+"_coco_format.json")) bit when trying to convert to coco-json throws this error NotImplementedError: Conversion from BoxMode 4 to 1 is not supported yet which is occuring cause we are trying to convert BoxMode.XYWHA_ABS to BoxMode.XWWH_ABS. Did you also get the similar error? I am using the same dataset as you. @SujoyDUhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSujoyDU&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C09fa8d83f5c94b7bf62908d82e96f95a%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637310573251236453&sdata=M7rQNV2OrSaxPcwcxz%2FY1iwvvl5lUw18al8g6mhvjMI%3D&reserved=0 did you find a workaround? Any help would be appreciated. Thanks guys.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%23issuecomment-662747452&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C09fa8d83f5c94b7bf62908d82e96f95a%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637310573251236453&sdata=18RCHeMIT4%2F%2BKxah4BXcgyvBr4so%2BnSgENYQrHlu4XY%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU5GOKZ2LOII6VXXX73R45Y2PANCNFSM4JABAQJA&data=02%7C01%7Cs.mander%40lancaster.ac.uk%7C09fa8d83f5c94b7bf62908d82e96f95a%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637310573251246442&sdata=RPxZouPJnpj3lDzlNap7rpqHprHuo%2BYr3BI2qio1eFQ%3D&reserved=0.

[SujoyDU]

Hi @karan-shr I kind of found a work around that. Detectron2 supports XYWHA_ABS to XYXY_ABS. So you can do that first and then convert it from XYXY_ABS to XYWH_ABS.

[junliu-cn]

Apologies for delay:

Here's a stripped back notebook from my dissertation - Its pulling in data I've stored on my Google Drive so you'll need to rewrite the dataloader before trying it.

https://colab.research.google.com/drive/1ijCf9LwJ3HbXMBFXi4CM6qd5IBXlh330?usp=sharing

Hi @st7ma784, can I use rotated bbox with mask?

[st7ma784]

I can't recall off the top of my head the exact answer to that, though I think so?? (I recall a bit of faffing to get it working) I'll reply later when I'm at my desk again to have a look and see if I got there.

On 30 Oct 2020 08:32, CVer-Kaggler [e-mail] wrote:

This email originated outside the University. Check before clicking links or attachments.

Apologies for delay:

Here's a stripped back notebook from my dissertation - Its pulling in data I've stored on my Google Drive so you'll need to rewrite the dataloader before trying it.

https://colab.research.google.com/drive/1ijCf9LwJ3HbXMBFXi4CM6qd5IBXlh330?usp=sharinghttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcolab.research.google.com%2Fdrive%2F1ijCf9LwJ3HbXMBFXi4CM6qd5IBXlh330%3Fusp%3Dsharing&data=04%7C01%7Cs.mander3%40lancaster.ac.uk%7C358af81a6ced4aa79bd008d87cae5922%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637396435542864512%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3QnDE%2FPFw51AX4R8e6DMv7HAHyhBepZhLHYfAgRJ5P0%3D&reserved=0

Hi @st7ma784https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fst7ma784&data=04%7C01%7Cs.mander3%40lancaster.ac.uk%7C358af81a6ced4aa79bd008d87cae5922%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637396435542864512%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yxpFKcOhy4R99IVmuRbsuMDyOUtRJD72ZM61cK58acU%3D&reserved=0, can I use rotated bbox with mask?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2Fdetectron2%2Fissues%2F21%23issuecomment-719412667&data=04%7C01%7Cs.mander3%40lancaster.ac.uk%7C358af81a6ced4aa79bd008d87cae5922%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637396435542864512%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ky7rXPB14JwbF8jUutXs71YkPLtwoajuAnSP07uEiLU%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAI25TU7HHPKC5EWEGYBIZWLSNJ2Z3ANCNFSM4JABAQJA&data=04%7C01%7Cs.mander3%40lancaster.ac.uk%7C358af81a6ced4aa79bd008d87cae5922%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C637396435542874469%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=sKGRWS7m70ibBORvgfceeN72yis%2Fdvnh76hsW8lh5D4%3D&reserved=0.

[annesteenbeek]

For anyone else still working on a rotated implementation, I've clean up the very helpful work from @st7ma784 to be a bit simpler to run.
Let me know if there are any improvements.
Hope it is of help to someone.

https://colab.research.google.com/drive/1324pVWKpyscblSNUU1aqSDuDJb6liAvo?usp=sharing