Evalai: Error in receiving message from submission queue with error 'NoneType' object has no attribute 'get'

[deshraj]

https://sentry.io/cloudcv/evalai/issues/498341967/

Error in receiving message from submission queue with error 'NoneType' object has no attribute 'get'

Note that this error occurs whenever you try to make submission to a challenge which is not active/not approved. This leads to making the submission status stuck to SUBMITTED state only. Instead we would like to change it to canceled and show user a readable error message that 'Sorry your submission cannot be processed since the challenge is not active or is not approved by EvalAI Admin.'

[alrlchoa]

May I try fixing this bug?

[RishabhJain2018]

@alrlchoa Sure go ahead!

[alrlchoa]

Hi @RishabhJain2018 !
Thanks! Only got to see this now. While looking through the code, I had a couple of questions:

1) For the approval part of the admin, I opened up the Model for the Challenges. Correct me if I am wrong, but I did not see any public function for calling approval property. Is that by design? If so, where may I get it? If not, may I create said public function or will something go wrong if I do (like security-wise or something)?

2) What should I do with the job/submission when it gets disapproved? Should I dump it or let them keep it in front-end (like if the job is minutes away of going active)?

Thanks!

[sashank27]

@deshraj @RishabhJain2018 I would like to add this functionality. I am not very thorough with the codebase yet, so please guide me on how to proceed with this.

[nikhilmudholkar]

@deshraj @RishabhJain2018 I would like to work on this issue.

[RishabhJain2018]

Hi @nikhilmudholkar Sure, you can work on this issue.

[nikhilmudholkar]

Can someone help me with some basic pointers for solving this issue? The entire codebase to new to me. At least, can anyone point me to relevant files and scripts where the issue needs to be solved?

[KhalidRmb]

@nikhilmudholkar Hi! This might be a good place to get started with understand the codebase: https://evalai.readthedocs.io/en/latest/submission.html

[RishabhJain2018]

Hi, @nikhilmudholkar Welcome to CloudCV! Assigning this issue to you.

In order to solve this issue, you have to add a check if the challenge is approved by an EvalAI admin or not in the API that takes in the submissions.
The API is written in jobs/views.py.

I hope this will help you. Feel free to ask if you have more questions :). I'll be happy to help.

[nikhilmudholkar]

Thanks, I am working on this issue. This is another little problem that I encountered while the development.
There is an example folder containing sample test_zip files and sample evaluation scripts. As you might be knowing that EvalAI takes metadata from zip_challenge.yaml file inside the test_zip_folder to create a new challenge. The start and end dates in the .yaml file are outdated. So if a new user when tries to create a new challenge, the challenge is already over when created. This creates a lot of issues, particularly while submission.
I think these dates need to be updated so that every new user does not need to update the folder and then zip it again. Also, it is not intuitive that one needs to update the file. It takes some figuring out.

[nikhilmudholkar]

I have worked on the issue and will be submitting a PR shortly. I just have one doubt.
Do we need to throw some sort of error when the submitting to a challenge that is approved, is active but is not yet published? Currently, I am assuming that one can submit to a challenge which has not yet been published.
@RishabhJain2018 can you please clarify this so that I can issue the PR?

[RishabhJain2018]

Hi @nikhilmudholkar For the first comment, please feel free to open a issue & fix it in a PR.

Do we need to throw some sort of error when submitting to a challenge that is approved, is active but is not yet published?

No, we allow challenge hosts to keep the challenge private and make submissions in order to test if everything is working fine or not.

[nikhilmudholkar]

@deshraj @RishabhJain2018 I have worked on this issue. I will PR shortly. I am sure that the travis CI build will fail. This is because of the following reason.
From a lot of testing and trying as you can see from the long list of commits above, I came to the conclusion that there is a small flaw when testing for this condition. I looked into the test_views of jobs file and found out that the approved_by_admin property of the challenge object is not set ( and the default value of which is FALSE). So when the tests for all other functions are run, the challenge object has approved_by_admin property as false. This won't give an error unless we write the condition for catching this problem(which is what I was trying to do precisely). Now when I wrote condition to deal with this problem, the tests failed because the challenge object in all the tests has this property as false. So the condition which I just wrote was executing instead of the ones that were supposed to be executed according to the tests.
To indeed confirm this, I manually set (approved_by_admin=True) in the challenge object in the test_views. The Travis CI build was not failing now.
Can you guys please look into this and give suggestions?
Thanks a lot

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed as there is no further activity. Thank you for your contributions.