Label-studio: Cloud storage slow with 1000+ tasks

[keesschollaart81]

Describe the bug
When using either GCP or AWS cloud storage BBox tasks labeling is very slow, in this report I'll stick with S3 but the behavior for Google is very similar. Loading the tasks list takes minutes.

To Reproduce
Steps to reproduce the behavior:

I'm migrating from another tool so I generate the JSON files for tasks and completions. I have 3 buckets now:

• labelstudio-tasks (containing 0.json, 1,json, .... 1300.json)
• labelstudio-completions (containing 0.json, 1.json, ... 1300.json)
• labelstudio-images (containing filename1.png, another-file.png, ... 1300thfile.png)

A tasks file (0.json) looks like this:

{
  "data": {
    "image": "s3://labelstudio-images/evidence_637431187722247603.png"
  },
  "id": 0
}

A completion file looks like this:

{
  "completions": [
    {
      "lead_time": 0,
      "result": [
        {
          "from_name": "label",
          "image_rotation": 0,
          "original_height": 480,
          "original_width": 480,
          "to_name": "image",
          "type": "rectanglelabels",
          "value": {
            "height": 19.4746828677309,
            "rectanglelabels": [
              "person"
            ],
            "rotation": 0,
            "width": 8.906873574686431,
            "x": 21.53292474344356,
            "y": 65.31989381413912
          }
        }
      ]
    }
  ],
  "data": {
    "image": "s3://labelstudio-images/evidence_637431187722247603.png"
  },
  "id": 0
}

I install label-studio and start it with the following command:

label-studio start my_project --init --source s3 --source-path labelstudio-tasks --source-params '{ \"use_blob_urls\": false, \"regex\": \".*\" }' --target s3-completions --target-path labelstudio-completions' 

The app start correctly and eventually after configuring and a few minutes waiting, it shows the tasks. When I open a tasks it loads quickly and shows the correct bounding box.

I added some debug logging statements in the s3.py to see what is going on. There I observed that it downloads all of the task and completions files a bunch of times. The more files, the long it takes of course.

Expected behavior
I would expect the tool to be me optimized in working with a larger (cloud based) dataset. Maybe an initial import (with progress indication?) keeping a local index of filenames and compare that? Given the pager in the tasks list, only downloading the files from the current page would already help a lot.

Environment (please complete the following information):

• OS: Both Windows and Mac, Both GCS and S3
• Both 0.8.2 and the [new-onboarding branch] @ 2057a591fa7138b07175e6ef8bcc842341384c5e

[makseq]

Could you try new-onboarding branch again without any filters and orderings? When they are disabled the engine must take only requested page from the storage.

[makseq]

And why do you use two storages for input? I think you can use labelstudio-images only.

Just enable "Generate task data with URLs pointed to your ..." checkbox for labelstudio-images and LS will generate json tasks automatically.

[keesschollaart81]

Awesome, so no filters/ordering, that makes sense! I had a pagesize set (stored in a cookie)

It took me some time to get it right (I had some mistakes in the completion file generation) but now it's much better. It still downloads all the completions once you load the tasks list for the first time. This seems to be because page==1 here at this line!

Indeed, I do not need to create the task files myself. Now I only have 2 buckets (much better):

labelstudio-completions (containing 0, 1, ... 1300)
labelstudio-images (containing filename1.png, another-file.png, ... 1300thfile.png)

I initialize the project with label-studio start my_project --init --source s3 --source-path labelstudio-images --source-params '{ \"use_blob_urls\": true, \"regex\": \".*png\" }' --target s3-completions --target-path labelstudio-completions --target-params '{ }'

For some reason, it only works when the completion files have the same name as their id so a completion like:

Completion file (JSON)

{
    "completions": [
        {
            "created_at": 1609429477,
            "id": 11057,
            "lead_time": 254.996,
            "result": [
                {
                    "original_width": 480,
                    "original_height": 480,
                    "image_rotation": 0,
                    "value": {
                        "x": 77.8464988425926,
                        "y": 57.470703125,
                        "width": 4.21459056712963,
                        "height": 15.682418258101851,
                        "rotation": 0,
                        "rectanglelabels": [
                            "person"
                        ]
                    },
                    "id": "d071c254",
                    "from_name": "label",
                    "to_name": "image",
                    "type": "rectanglelabels"
                }
            ]
        }
    ],
    "data": {
        "image": "s3:\/\/labelstudio-images\/filename.png"
    },
    "id": 1008
}

... the file must be named 1008 (not 1008.json), is that to be expected?

[makseq]

This seems to be because page==1 here at this line!

Yes, we need to calculate total task and completion number in DB somewhere. And it will be done while retrieving the first page.
I'm talking about this:

[makseq]

the file must be named 1008 (not 1008.json), is that to be expected?

It seems to be a bug :slightly_smiling_face:

[keesschollaart81]

Yes, we need to calculate total task and completion number in DB somewhere. And it will be done while retrieving the first page.
I'm talking about this:

I see but, for the total-number, listing the files in the bucket should be sufficient? Now it's downloading all the completion files and for 1000s of files, that can take a few seconds up to more than a minute...

[makseq]

Yes, it's not good. But I don't figure other way out to get all completion count yet... We need to read them all.
But maybe we can show total task/completion number optionally.

[keesschollaart81]

I leave that up to you, it sounds like a nice little improvement! Thanks for the prompt help/support! Feel free to close this issue, or to leave it open to keep track of for this bug/feature-idea!

[makseq]

I will add this option to the project config. Or you can do it via PR, it will be faster I think :slightly_smiling_face:

[makseq]

@keesschollaart81 Please, check the last commit from feature/new-onboarding branch. Add to your project/config.json this line

"first_page_full_render": false

[keesschollaart81]

Yes! Perfect, now it's fast for the first load and also while using it after that!

To get back to my question:

listing the files in the bucket should be sufficient?

By looking at the code, I now understand why the answer is: no because a single completions-file can contain multiple completions. So this boils down to the difference between (a) the number of tasks having at least 1 completion and (b)the total number of completions over all tasks. The current implementation is (b). For (a) you would not need to read the contents of all the completion files, whereas for (b) you would. I would be happy with (a) but I guess there are scenario's where (b) is important???

In any case! Thanks!!

[makseq]

single completions-file can contain multiple completions

Yes, you're absolutely right. We will switch our engine to sqlite database in the nearest future, so I hope these problems will be passed away.