Turicreate: Unclear how to set up annotations column for Object Detection

[ProjectDent]

The guide for Object Detection doesn't cover setting up the data, including how to attach the "annotations" category.

What I've done is to create a separate annotations file, which looks like this:

{
  "1.jpg": {
    "type": "rectangle",
    "coordinates": {
      "height": 97,
      "width": 243,
      "x": 4224,
      "y": 1821
    },
    "label": "cw"
}

I then setup my data using load_images, and this file:

# Load images
data = tc.image_analysis.load_images('train', with_path=True)
# Open annotations file as dict
annotations = eval(open("annotations").read())
# Add annotations column to SFrame, using the annotations dict key with the same name as the file name
data["annotations"] = data["path"].apply(lambda path: bounds[os.path.split(path)[1]])

That works well, and if I print data, I get something like this:

+-------------------------------+---------------------------+
|              path             |           image           |
+-------------------------------+---------------------------+
| /Users/Andrew/Code/turi/cw... | Height: 3816 Width: 11056 |
| /Users/Andrew/Code/turi/cw... | Height: 3888 Width: 10672 |
| /Users/Andrew/Code/turi/cw... |  Height: 3656 Width: 9700 |
| /Users/Andrew/Code/turi/cw... |  Height: 3872 Width: 8280 |
+-------------------------------+---------------------------+
+-------------------------------+
|          annotations          |
+-------------------------------+
| {'type': 'rectangle', 'coo... |
| {'type': 'rectangle', 'coo... |
| {'type': 'rectangle', 'coo... |
| {'type': 'rectangle', 'coo... |
+-------------------------------+

I don't know why that's separated onto 2 lines in the console - likely just for sizing reasons.

So then I get to this line in the Object Detection guide, where it intends to visualise the annotations applied to the data:

tc.object_detector.util.draw_bounding_boxes(data["image"], data["annotations"])

When I run this, I get this error in the console:

Traceback (most recent call last):
  File "app.py", line 62, in <module>
    load_data(bounds)
  File "app.py", line 23, in load_data
    tc.object_detector.util.draw_bounding_boxes(data["image"], data["annotations"])
  File "/Users/Andrew/turi/lib/python2.7/site-packages/turicreate/toolkits/object_detector/util/_visualization.py", line 139, in draw_bounding_boxes
    .apply(draw_single_image))
  File "/Users/Andrew/turi/lib/python2.7/site-packages/turicreate/data_structures/sframe.py", line 2463, in apply
    dryrun = [fn(row) for row in test_sf]
  File "/Users/Andrew/turi/lib/python2.7/site-packages/turicreate/toolkits/object_detector/util/_visualization.py", line 124, in draw_single_image
    _annotate_image(pil_img, anns, confidence_threshold=confidence_threshold)
  File "/Users/Andrew/turi/lib/python2.7/site-packages/turicreate/toolkits/object_detector/util/_visualization.py", line 49, in _annotate_image
    for ann in reversed(anns):
TypeError: argument to reversed() must be a sequence

In addition, if I comment that out, and then go ahead and do:

model = tc.object_detector.create(data, feature="image", annotations="annotations")

I get the error:

Traceback (most recent call last):
  File "app.py", line 65, in <module>
    learn()
  File "app.py", line 37, in learn
    model = tc.object_detector.create(data, feature="image", annotations="annotations")
  File "/Users/Andrew/turi/lib/python2.7/site-packages/turicreate/toolkits/object_detector/object_detector.py", line 170, in create
    require_annotations=True)
  File "/Users/Andrew/turi/lib/python2.7/site-packages/turicreate/toolkits/object_detector/object_detector.py", line 66, in _raise_error_if_not_detection_sframe
    raise _ToolkitError("Annotations column must contain lists")
turicreate.toolkits._main.ToolkitError: Annotations column must contain lists

Presumably I'm setting up my annotations column incorrectly to what its' expecting.

[gustavla]

Thanks for describing your pain points in great detail! We know that setting up the data for the object detector is tricky and appreciate all the feedback we can get to help make the process easier.

It looks like you are all set, except for one thing. Each cell in the annotations column must contain a list of dictionaries. This is to allow more than one object per image. Of course, if you are only planning to have one object per image, that is perfectly fine too and I have trained successful detectors on such data. However, we still need it to be a list in that case. You can correct this for your data by calling:

data['annotations'] = data['annotations'].apply(lambda x: [x])

This will put what you have now into lists of length 1 and the detector functions should be happy.

[johnyquest7]

Thank you for explaining it. I was also running into the same problem.

[gustavla]

@johnyquest7 Thanks for the feedback! We'll try to make this clearer in the documentation and perhaps also in the error message.

[atthaje]

The annotation column in my SFrame ends up having 3 rows after I read annotation list for 3 objects in an image, but the image column has only 1 row representing the corresponding image, thus resulting in row mismatch. Shall I be loading 3 copies of the image if I intend to detect 3 objects in an image such that number of rows match ?

[gustavla]

@atthaje If you have an image with three objects, then it should be 1 row in an SFrame and the annotations column should be a list of three dictionaries. I hope that clarifies things! Feel free to open up a new issue if you are still having problems.

[atthaje]

Thank you for the clarification.

[znation]

Reopening to track a product change to eliminate this pain point: let's allow for the annotations being either a single dictionary or a list of dictionaries.

[shantanuchhabra]

This was fixed by #293