Parlai: AWS and Heroku integration: Paid?

[kushalchawla]

Hey. I am exploring ParlAI for dialog data collection between two MTurkers. I was able to host the sample qa_data_collection task in a sandbox environment for free using AWS and Heroku. I am wondering about where exactly does ParlAI require integration with AWS and Heroku. For running a large-scale data collection study, what payment plans do I need to set up for both of them, apart from the usual payments which go to MTurkers on AMT?

Apologies, if this is covered somewhere in the documentation. I tried but could not find anything directly related.

Thanks.

[stephenroller]

For data collection I believe you only need Heroku. We tend to use the unpaid tier when debugging, and then switch to the hobby tier (using --hobby) when actually running collections. It's very modestly priced, especially if you take care to check it's shut down afterwards. Nonetheless, you can continue to use the free tier during live collection, you should just lower the number of simultaneous workers to minimize load.

Flagging @JackUrb if he has additional comments.

[stephenroller]

(Also only the ParlAI MTurk parts require heroku. The model training components don't require any third party services, unless you need to rent GPUs or similar)

[kushalchawla]

Hey @stephenroller

In reference to your earlier comment (copied from above):
"For data collection I believe you only need Heroku. We tend to use the unpaid tier when debugging, and then switch to the hobby tier (using --hobby) when actually running collections. It's very modestly priced, especially if you take care to check it's shut down afterwards. Nonetheless, you can continue to use the free tier during live collection, you should just lower the number of simultaneous workers to minimize load."

What is the usual number of simultaneous workers which I can allow while using the hobby tier? Like an expected load which the system can take up?

The default value for this parameter in ParlAI is 30, which I have been using with the free version up until now. For the live study, we are planning to use the hobby tier, as per your suggestion. But if we can increase this number now, that would be great.

Please let me know asap.

[stephenroller]

It depends on the heaviness of what you're doing. If it involves talking with a model, then you're limited to what sort of simultaneous throughput your hardware can handle of that model. If it involves pairing users, then the computer isn't doing much, and you can crank it up.

30 seems like a pretty reasonable level to me. My experiments often run with like 20, but I don't do a ton of Turking.

@jxmsML @emilydinan @EricMichaelSmith maybe have other recommendations.

[kushalchawla]

Hey @stephenroller.

Yes, it involves pairing up MTurkers for like 10-15 minutes or so. The backend is just involved in sending out System messages every now and then and finally storing the logs at the end.

I am looking for an approximate number that I can crank it up to. Or is it something which I should just try and see? In the latter case, how to "see"?

Hoping to get some more recommendations from others. Thanks a lot.
I think increasing this will help us reach a wider MTurk pool right. -> This can really help in getting the data quickly.

[stephenroller]

Try 100? That has seemed safe for me. The thing to really do is monitor with htop or top or whatever. If all your cores are overloaded, then you need to cancel the run and lower it. If your machine is staying nice and cool, then you shouldn't have much problem.

[kushalchawla]

Hey @stephenroller

So it is about my machine and nothing to do with Heroku? Can hobby handle all this?

[stephenroller]

I think it depends on both, but heroku is mostly acting as a simple proxy. I don't think hobby should have any problem with 100 simultaneous connections. Free tier does tend to get you de-prioritized tho, so it can have trouble maintaining smooth connections.

[kushalchawla]

Oh got it. Thank you so much. I will try that out.

[Stefan1220]

Hi guys,

I am also using ParlAI for mturk task (which would involve a local dataset and a worker), and have a question about Heroku cost. For the Hobby tier, I checked the Heroku website, and it says $7 per dyno per month for each application. In my setting, each link(application) that I launch using heroku will contain 5 HITs. But I am not quite sure about the general dyno usage in turk task, do you have any idea that how much should I expect the cost if I push a mturk link live using the Hobby tier?

Thanks!

[JackUrb]

If you're only launching one MTurkManager run at a time, that task will only require one dyno, and will only be active during the time you're running the task.

[Stefan1220]

Thanks for the quick reply!

So does that mean if I only use MTurkManager to launch one link, then it only require one dyno per month (that is $7), no matter how many HITs in that link and how many workers will try that link?

[JackUrb]

This is at least in line with our usage.

[stephenroller]

And to clarify, we expect you will only run that dyno for the time you are collecting the data. So if you finish collecting in 1 day, it will cost $7 / (30.5) = $0.23, far far less than you will be paying the annotators.