Label-studio: Distinct collaborators receiving the same task

[rodrigocaus]

Describe the bug
When two collaborators log into the same project they receive the same task
Backend indicates an error:

[2021-04-12 22:11:03,640] [tasks.models::set_lock::103] [ERROR] Current number of locks for task 2 is 1, but overlap=1: that's a bug because this task should not be taken in a label stream (task should be locked)

In the user screen that logged last, sometimes, the "Submit" button disappear.

To Reproduce
Steps to reproduce the behavior:

1. Create accounts "user1@localhost" and "user2@localhost"
2. Log user1 and create a sample project
3. Import file with multiple tasks
4. Click to label a task with user1
5. Log user2 (using another browser or anonymous session)
6. Click to label a task with user2

Expected behavior
Users receiving different tasks each.

Screenshots
Click to label

user2 disappearing top toolbar

Environment (please complete the following information):

• OS: Ubuntu 20.04 focal
• Label Studio Version 1.0.1

EDIT
Better description to steps to reproduce the issue and possibly related "Submit button disappear" issue reported above and by @bolotyuh

[makseq]

@rodrigocaus

Click on "..." button to label the project

Do you mean blue Label button on Data Manager ?

[bolotyuh]

I have the same problem. I think the reason in https://github.com/heartexlabs/label-studio/blob/4f3df84fd38872c6a2405a2d8e330076bebf22db/label_studio/projects/api.py#L432

In this case, using just the first sample from data manager queryset
https://github.com/heartexlabs/label-studio/blob/4f3df84fd38872c6a2405a2d8e330076bebf22db/label_studio/data_manager/actions/next_task.py#L19

[rodrigocaus]

@makseq

The blue Label button also applies. But what I meant was this Label button:

I think they do the same.

[makseq]

@bolotyuh @rodrigocaus
The problem is not so trivial as it might seem. When LS generates the next task for the annotator, it must lock this task for a while until the annotator does his job. And this duration is unpredictable: it could be 1 second, 1 minute, 1 hour, etc.

We tried to use task locks and the label stream looks like what you expect: annotators are given different tasks all the time. Although if you try to press Label again, you won't see unlabeled & locked tasks, because they have been locked by some annotators for several hours in edge cases. And it's confusing.

So, you can have two fast workarounds:

• Rewrite this line
  https://github.com/heartexlabs/label-studio/blob/4f3df84fd38872c6a2405a2d8e330076bebf22db/label_studio/projects/api.py#L434 to next_task = not_solved_tasks.order_by('?').first(), this will randomize tasks on the each API call. It solves the problem particularly.

• Create individual tabs for the each annotator and split tasks via filter id ranges.

[bolotyuh]

@makseq Thanks you really help me.

[bolotyuh]

@makseq How is used the parameter sampling I thought it resolve this type of problem(randomize tasks)?

[makseq]

@bolotyuh It seems sampling is skipped always because of Data manager tabs usage.

You can remove this if block to enable it back:
https://github.com/heartexlabs/label-studio/blob/4f3df84fd38872c6a2405a2d8e330076bebf22db/label_studio/projects/api.py#L433

[bolotyuh]

I did it but get a new problem when disappearing top toolbar

I realized if you want to turn back you need to delete anotation

[makseq]

Yep, once I saw this problem, but can't reproduce it.

[bolotyuh]

Yep, once I saw this problem, but can't reproduce it.

When you refresh the page on the same task top toolbar disappearing as I understood it because we already have annotation

[makseq]

@bolotyuh Trying to reproduce it, but still can't..

[rodrigocaus]

@makseq thank you for your answer. Using something like:

# ordered by data manager
if external_prepared_tasks_used:
    next_task = next((t for t in not_solved_tasks.all() if not t.has_lock()), None)
    if not next_task:
        raise NotFound('No more tasks found')
    return self._make_response(next_task, request)

I can "simulate" a filter on not_solved_tasks (something like not_solved_tasks.filter(has_lock=False).first()). Since the task lock seems to expire, eventually unsolved tasks can be retrieved again later. Could this work?

[makseq]

Yes, it might work. Although this block should also be taken into account too:
https://github.com/heartexlabs/label-studio/blob/4f3df84fd38872c6a2405a2d8e330076bebf22db/label_studio/projects/api.py#L439

[rodrigocaus]

I see, my approach has this problem: Same User locks different tasks unnecessarily, so the project can falsely run out of tasks quickly (Users can lock as many tasks as they want as soon as they click on Label button)

I can suggest a similar solution:
```

ordered by data manager

if external_prepared_tasks_used:
try:
next_task = next((
t for t in not_solved_tasks.all()
if t.locks.filter(user=user).count() > 0
))
return self._make_response(next_task, request, use_task_lock=False)
except StopIteration:
next_task = next((t for t in not_solved_tasks.all() if not t.has_lock()), None)
# could be next_task = not_solved_tasks.first()
if not next_task:
raise NotFound('No more tasks found')
return self._make_response(next_task, request)
`` It is not pretty and it might have a better solution using filters onnot_solved_tasks`, but it seems to solve the problem you pointed.