Parlai: Cannot access link to HIT mturk examples

[ntanhhus]

As the title, I've tried to execute the example code in mturk folder, the execution is fine but when I access the link to HIT, it says:

This HIT requires Qualifications
This Requester has specified Qualifications for this HIT. At this time, you do not meet those Qualifications.

I've checked the agent_qualifications that passed to mturk_manager, it is empty so there should be no qualifications required, right? Am I missing something here?

Here is the console log when I execute the code:

[ optional arguments: ]
[ ag_shutdown_time: 120 ]
[ allowed_conversations: 0 ]
[ block_qualification: None ]
[ count_complete: False ]
[ datapath: /home/ubuntu/tanh/ParlAI/data ]
[ disconnect_qualification: None ]
[ hard_block: False ]
[ heroku_team: None ]
[ is_debug: False ]
[ is_sandbox: True ]
[ local: False ]
[ log_level: 20 ]
[ max_connections: 30 ]
[ max_hits_per_worker: 0 ]
[ max_persona_time: 300 ]
[ max_resp_time: 150 ]
[ max_time: 0 ]
[ max_time_qual: None ]
[ max_turns: 10 ]
[ min_messages: 0 ]
[ min_turns: 5 ]
[ mturk_log_path: /home/ubuntu/tanh/ParlAI/logs/mturk ]
[ num_conversations: 1 ]
[ persona_type: both ]
[ personas_path: None ]
[ range_turn: 5,7 ]
[ revised: False ]
[ reward: 0.05 ]
[ task: None ]
[ unique_qual_name: None ]
[ unique_worker: False ]
[ verbose: False ]

You are going to allow workers from Amazon Mechanical Turk to be an agent in ParlAI.
During this process, Internet connection is required, and you should turn off your computer's auto-sleep feature.
Enough HITs will be launched over time up to a max of 1.5 times the amount requested until the desired number of conversations start.
Please press Enter to continue...

Querying the parlai website for possible notices...
Setting up MTurk server...
Heroku: Collecting files...
Heroku: Starting server...
› Warning: token will expire 09/28/2019
› Use heroku authorizations:create to generate a long-term token
Creating ubuntu-3651e5cb-personachatcha... done
Enabling http-session-affinity for ubuntu-3651e5cb-personachatcha... done
Scaling dynos... done, now running web at 1:Free
MTurk server setup done.

Botocore couldn't subscribe to HIT events, perhaps you tried to register to localhost?
Creating HITs...
You should be calling ready_to_accept_workers before create_hits to ensure that the socket is connected beforehits are added. This will be enforced in future versions.
Link to HIT: https://workersandbox.mturk.com/mturk/preview?groupId=3LOAKXD6JDBSZ9YK2DEDUALJZNXERQ

Waiting for Turkers to respond... (Please don't close your laptop or put your computer into sleep or standby mode.)

Local: Setting up WebSocket...
WebSocket set up!

Any help would be appreciated, thank you.

[JackUrb]

Hi @ntanhhus - the only thing I can think of is that MTurkManager defaults to using english speaking locales as the setup isn't really prepared for other languages, so it uses a whitelist of certain specific countries. If this is the issue, you can disable this for testing by providing a location qualification of your own that whitelists everything or at least wherever your account is listed as being located.

[ntanhhus]

Hi @JackUrb, thank you so much for the suggestion, it worked!

The location qualification that I added:
[{'QualificationTypeId': '00000000000000000071', 'Comparator': 'EqualTo', 'LocaleValues': [{'Country': 'VN'}]}]

[FMCumhaill]

Hey @ntanhhus might be able to explain how you implemented the specific location qualification?

[JackUrb]

View the qualification_flow_example Task for more context. You end up passing a qualification array to one of the MTurk manager's lifecycle functions to use custom qualification settings

[JEM-Mosig]

Thanks @ntanhhus for the suggestion. Unfortunately, it doesn't work for me. In the qa_data_collection/run.py script, I have replaced mturk_manager.create_hits() with

agent_qualifications = [
            {
                'QualificationTypeId': "00000000000000000071",
                'Comparator': 'EqualTo',
                'LocaleValues': [{'Country': 'DE'}],
            }
]
mturk_manager.create_hits(qualifications=agent_qualifications)

but the problem remains. I used 'DE' for the country code, since I am based in Germany. I also tried NotEqualTo and VN (since I am not in Vietnam) without success. What else can I try?

[JEM-Mosig]

Ah, I figured it out. In sandbox mode, ParlAI ignores the qualifications that i set: https://github.com/facebookresearch/ParlAI/blob/0456d2ae7964e6914275633ef422499fb6592d9c/parlai/mturk/core/mturk_manager.py#L1649
Thus, even if I specify EqualTo, DE, ParlAI only allows english speaking countries in Sandbox. I removed that code block from the ParlAI source and it works now.