Evalai: Add support in submission worker to update file field changes for Challenge and Challenge Phase

[taranjeet]

__Problem__

If the Challenge Host updates any of evaluation_script or test_annotation file, then there is no way to propagate that change to the worker.

__Please be more clear__ ?

Presently the submission worker when starts fetches all the active challenges and loads evaluation_script for each challenge. It also fetches active challenge phase for each challenge and loads test_annotation for each phase. We need to extend that code to add support for handling update requests for these files.

__Why__ ?

Imagine a case where in a Challenge Hosts whose challenge is active and hence running, wants to update evaluation_script for a challenge. He logs in and updates the file but still sees that submission are being processed using the older version of evaluation_script. He wonders why is this happening ??

So we want to help him and add support to propagate the changes done by him to the worker, because it is the worker who is doing the heavy task of processing submission.

__Impact__ ?

It actually guarantees that whenever any submission is evaluated, it is evaluated against the write set of parameters.

__How__ ?

Overall, this will involve refactoring scripts/workers/submission_worker.py. The concerned function which extracts challenge and phase data presently is extract_challenge_data. This function may be modified or a new function may be added. This will actually require some thoughts from the contributor.

One very important thing which needs to be pointed out is how the worker will detect that the file is changed. Here RabbitMQ comes into action. If you see add_challenge_queue in scripts/workers/submission_worker.py, it binds on key challenge.*.*. The first * here will help us decide what the intent of the message is.

Whenever any new challenge will be added, you can assume that challenge.add.* key will be there and whenever any update is done, you can assume that challenge.edit.* will be published.

So you can now add checks according to the key and then code accordingly.

For now being you can assume that keys published will be challenge.add.* and challenge.edit.*.

__TakeAways__

About how the submission worker works in detail, RabbitMQ and its related terminologies. This issue will provide you more depth about RabbitMQ and how queue function.

Also I will prefer that before coding, its better to discuss the approach over here only, so that we can get an idea and update if something is not right.

[mayank-agarwal-96]

@TroJan Can I work on this?

[taranjeet]

@mayank-agarwal-96 : sure :)

[taranjeet]

@mayank-agarwal-96 : Let me know if you are struck somewhere. Will be happy to help :)

[mayank-agarwal-96]

@TroJan : sure :) I am currently reading about RabbitMQ, as I am not experienced with it. I will ask if I get stuck.

[mayank-agarwal-96]

@TroJan I was reading this pull request created by you for documentation about workers. I noticed that all the evaluation scripts are loaded in a dictionary before a worker fully starts. So whenever a challenge.edit.* is published, we need to update the EVALUATION_SCRIPTS dictionary? Also what needs to be done about previously evaluated submissions?

[taranjeet]

@mayank-agarwal-96 : Yeah whenever a message for challenge.edit.* is published, you need to update EVALUATION_SCRIPTS dictionary. If you clearly see the structure of the dictionary, it uses challenge id as its keys. Challenge id has to be available in the message and then you can dynamically import the module and then reload the module in EVALUATION_SCRIPTS.

As of now, we are going with the fact that we wont do anything for submissions which are already submitted and evaluated. Hence no need of considering them.

[taranjeet]

@mayank-agarwal-96 : is this in progress? If you are struck somewhere you can ask it here!

[mayank-agarwal-96]

Yes I am still trying to fix this one. I just wasn't able to dedicate proper time. I'll keep posting if I feel stuck.

[taranjeet]

@mayank-agarwal-96 : any update ?

[mayank-agarwal-96]

@TroJan As of now, I am not able to work on the issue because of time constraints. Please reassign if someone wants to take this up. Sorry for the delay.

[yadavankit]

@TroJan @mayank-agarwal-96 is this taken?

[deshraj]

@yadavankit I think @mayank-agarwal-96 is busy with his exams. So, if you are interested, then let me know. I can assign you the issue.

[yadavankit]

@deshraj yeah sure, am interested in doing this one :)

[deshraj]

Assigning it to you. :)

[mohitanand001]

@taranjeet @deshraj . Hello, can this issue be assigned to me. I have read the docs regarding the worker, and it has made some sense. I'll try to clear it within 4-5 days and come up with a working solution.
Thanks

[mohitanand001]

@taranjeet @deshraj . I have read the docs and have drawn some conclusions, please rectify if I'm wrong.

1. apps/jobs/sender.py behaves as the sender/producer for the RabbitMQ broker (which is called by API).
2. the main function in submission_worker.py behaves as the receiver/consumer.
3. In the docs it is mentioned that submission_worker.py works as a daemon. Does this mean it runs continuously in the background? Please clarify.
4. Lastly,since the challenge can be added and edited from admin panel only (as of now) , how do I emulate/demonstrate challenge.add.* and challenge.edit.*.

Please clarify.
Thanks

[taranjeet]

yeah on production environment, we run it as daemon process. On development environment, we simply run it as a python process.

Regarding the challenge.add.* and challenge.edit.*, you will need to write a small script yourself, that creates a connection and then publishes a message. This part is actually missing. We are planning to write test case for it, so that it can be completed and tested end to end.

[mohitanand001]

@taranjeet @deshraj Okay I've a solution (just a prototype). I'll create a script similar to jobs/sender.py which will have two functions challenge_add_publish_message and challenge_edit_publish_message. Since, currently there is no view to invoke these functions, they'll be invoked using command line arguments and they will route the message to the add_challenge_queue . Is this the right approach ?

[yadavankit]

@farziengineer yeah, I too have followed kinda same approach to cater this issue, a better way to invoke the message publishing functions for now would be using post_save signal from Challenge with created as False. This would invoke the message publishing function whenever Challenge details are changed (which can be done from superuser dashboard for now). I will be adding a WIP PR asap.

[mohitanand001]

@yadavankit Sure :+1:

[guyandtheworld]

@taranjeet @yadavankit Can I take over?

[deshraj]

@isht3 feel free to take this up.

[guyandtheworld]

@deshraj Okay, thanks!

[taranjeet]

@deshraj @RishabhJain2018 : this one also. Can we please close it if not required?

[deshraj]

Yeah I agree. We are moving towards using AWS SQS instead of RabbitMQ and hence we won't need this now. Closing this issue.