Evalai: Add a feature to re-reun a submission by challenge host using UI

[RishabhJain2018]

Current Behaviour

If a challenge host wants to re-run a submission then the host has to email EvalAI admin to re-run a particular submission from admin.

Expected Behaviour

The challenge host should be able to re-run the submission from the UI.

Please feel free to post the approach first.

[yashdusing]

@RishabhJain2018 I'd like to work on this

[RishabhJain2018]

@yashdusing You can take this up.

[KhalidRmb]

@RishabhJain2018 I'd like to work on this as well.

[yashdusing]

@RishabhJain2018 also, could you please verify the doubts I've asked on gitter please

[RishabhJain2018]

Sure! I'll take a look at it soon.

[yashdusing]

@RishabhJain2018 So, the challenge host no longer has the access to the annotation file he/she previously submitted. So, the challenge host approaches the admin to fetch the submission annotation file to re-run it on the challenge ? Is that right or am I missing something ?

[RishabhJain2018]

So, the challenge host no longer has the access to the annotation file he/she previously submitted. So, the challenge host approaches the admin to fetch the submission annotation file to re-run it on the challenge ? Is that right or am I missing something ?

I'm not sure why you're talking about annotation file here?

[yashdusing]

@RishabhJain2018 the file submitted by the participant which the challenge host wants to re-evaluate (re-run)

[yashdusing]

@RishabhJain2018 How about the challenge host having a re-run submission button on the leaderboard itself ?
Also, should I use the existing run_submission function in the workers folder or write a function to get the existing files which might be saved onto the temp directory when worker is initiated ?

[RishabhJain2018]

Hey, @yashdusing The file submitted by the participant is the user_submission_file. Please don't confuse it with test_annotation_file.

How about the challenge host having a re-run submission button on the leaderboard itself ?

No, since the leaderboard doesn't list all the entries (as participants can make them private), I think it'd be ideal to have it under view all submissions tab.

Also, should I use the existing run_submission function in the workers folder or write a function to get the existing files which might be saved onto the temp directory when worker is initiated ?

No, please try to understand the architecture, how it works. You need to pass the submission message to the SQS queue and it'll be processed automatically. I've already implemented an admin feature for it. Please take a look at that.

[Ayukha]

@yashdusing I guess publishing the submission parameters to the submission Queue would do the work, a similar kind of approach that we used here.
https://github.com/Cloud-CV/EvalAI/blob/343089d6b246bd0d5815ad9bb7288a958195211a/apps/jobs/admin.py#L57-L70

[RishabhJain2018]

Yes @Ayukha . Thanks for pointing to the code.

[yashdusing]

@RishabhJain2018 Thanks for clarifying. I was a little confused and thought I needed to make changes directly to worker.py 😅

[yashdusing]

@RishabhJain2018 could you please check my PRs since most of them are not reviewed. Also, my proposal as well

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed as there is no further activity. Thank you for your contributions.