Label-studio: With prediction-score-min strategy, no labeling tasks are found

[andreaferretti]

Describe the bug
I tried to setup an active learning environment, where the interface always suggests the labeling task on which the model is most uncertain. To do so, I connected an ML backend and ran label-studio with "sampling": "prediction-score-min". Now I cannot label anything

To Reproduce
Steps to reproduce the behavior:

1. Create and connect a ML backend
2. Start Label-studio with "sampling": "prediction-score-min"
3. Import some tasks
4. Go to labeling page: no labeling tasks are found

Expected behavior
I would expected to be presented with labeling tasks, in increasing order of model prediction score.

Screenshots

[makseq]

@andreaferretti are you sure that tasks have predictions?

[andreaferretti]

They don't. Shouldn't predictions be created on the fly? When I choose sequential method, whenever I open a new labeling task, I see a prediction - I assume that the frontend queries the model on the fly.

What should I do to ensure that the ML backend is called on the existing tasks, so that labeling tasks show up here?

By the way, if there are no predictions, it would be better to show some labeling task at random, instead of nothing

[fromnorthpole]

@andreaferretti when you are using prediction-score-min mode for sampling, it expects that every task in a dataset has a prediction, then selecting the task with minimal prediction score. Connected ML backend provides only prediction for the currently selected task. You can still create predictions for the entire dataset using your ML backend by:

1. Specifying "enable_predictions_button": true in project/config.json
2. Clicking on "Start Predictions" button on the Model page

Then, after all predictions are completed, the prediction-score-min sampling method should work as expected.

But your suggestion about sampling fallback is definitely reasonable, we'll include it in a future version

[andreaferretti]

Thank you, now it is more clear

[makseq]

@andreaferretti can we close this issue? or you have more questions?

[andreaferretti]

It is ok to close, thanks