Parlai: Deploying chat service to AWS or Heroku

[jakeane]

Hi all,

I swear the solution to this is staring me right in the face, but I definite answer from you is likely better in the long run.

I have been part of a team with @mvh57 that is developing a custom chat service through your platform, and we have been interacting with it through the "Browser" method specified on your documentation. However, we want to bring take this chat service to a remote server hosted on AWS (or Heroku), so multiple users can have one-on-one conversations with the chat service through the client application we have also built. However, I am not sure how to proceed with this.

It seems like the file chat_service/core/server/server.js is a potential entry point, but I wonder if the chat_service/services/browser_chat/run.py file we have previously used has any part in this solution. So, I suppose the question here is this: Given a chat service we have successfully run through a local browser, which is based on a custom model, how can this be deployed onto AWS or Heroku? Such that multiple users can interact with it one-on-one?

Let me know if there are any additional bits of context that you need to answer this question. Otherwise I really appreciate you all for this platform.

-Jack

[stephenroller]

@klshuster

[stephenroller]

And @JackUrb

I know they have deployed to AWS before for an external demo.

[JackUrb]

Hi Jack - hard to know quite what you're trying to do here - is the goal to just put the model onto the AWS server and launch the client there, or to have a proxy setup where heroku or aws routes the incoming requests to the server running ParlAI?

For the former, you should just be able to just run the same script on the remote server and access it at at the public address and port. (If you want that to be addressable without port, you'll need to set up some reverse proxy to route 80 to the launched port, or run python as admin).

For the latter, it's a somewhat complicated setup and we haven't incorporated the router setup that ParlAI MTurk used into the chat service implementation.

[jakeane]

Hi @JackUrb. Thanks for getting back so quickly.

So my goal is similar to the former you mentioned. We are trying to put a chat service onto a server through AWS, which would then be accessed through a separate client application. However, my backend and AWS knowledge is quite limited, so if you could describe the process in more detail, that would be great!

[JackUrb]

I'm by no means an expert on this stuff, just got it set up for a different project. This guide covers a lot of the steps if you follow up through "Opening your webpage". Unfortunately the rest of that guide is setting up nginx for another purpose, so other guides can cover the part of configuring it as a reverse proxy. The "Configure nginx as a reverse proxy" step of this guide may prove helpful. There are other resources online about this that will be better than the advice I can give.

[klshuster]

One thing I will contribute to this is that we have the websockets chat service implementation designed specifically to work with external servers - note that the browser_chat is actually just an implementation fo the websockets service that utilizes a local browser, however websockets should be able to handle most of the work here

[Parvez2017]

Hi all,
I am also facing similar issues with deployment on AWS. Can I use a pre-trained model like blenderbot on AWS and get response through a webhook which messenger is using?

Thanks in advance

[jakeane]

@JackUrb thank you so much for the help. Went through the guides you provided, and they more or less helped me get there! Also learned a lot through this process.

For those who will inevitably come across this thread in the future:

• Follow this guide (same one as above) until you reach the "Installing NGINX Software" section.
  
  
  
  • One important note is the instance type you select. The free type, "t2.micro" may not provide enough RAM to run the program, so you likely will have to go to larger. I picked "t2.large" for what its worth.
  
  
  • Another difference I did here was within "Configure Security Group" (step 6 of setting up AWS). I added an additional rule where type was "custom TCP", the port was 8080, and the service was anywhere.
  
  
• Assuming you are connected to the instance, transport your ParlAI files in. Install pip and python3-dev. One thing to note though:
  
  
  
  • The ParlAI directory tends to be bloated (sorry devs), so I would recommend trimming off as much fat as possible. This will allow you to install other modules (PyTorch was giving me memory space issues for a bit. Would recommend install it before transferring ParlAI).
  
  
• From there start your script using port 8080. Use PM2 to run in background.

[stephenroller]

We should link to this post from the docs.

[jakeane]

Hi all, again,

I found an issue with the setup I detailed above. If the chatservice is left running for an extended period of time, ~1-2 days, it will become non-responsive, as in not replying to any incoming messages. The "band-aid" solution for this is to set up a cron job to restart the chatservice every 8 hours, which has not let to any issues yet as we just have a small test group using the app. The solution is exactly as follows:

• From the EC2 instance, type crontab -e. You may have to set some preferences like what editor to use for it.
• At the bottom of the file, add 0 */8 * * * /usr/local/bin/pm2 restart chatbot. Then save and quit.

However, this would clearly not scale, and it would inevitably lead to some bad experiences for users. I am not sure if the root cause of this issue lies on AWS or ParlAI or somewhere in between. I know the setup I currently have, detailed above, is kind of 'hacky', so maybe I have to set up the Websocket API in the AWS API Gateway for more 'proper' usage. If you guys have any knowledge on keeping the chatbot websocket service running over long periods of time, let me know.

There is also another issue going on here, when a user connects with the websocket, starts having a conversation, and then stops for a brief period of time, ~5-10 minutes, the chatbot becomes unresponsive to the user. Currently I restart the page in order to restart the websocket connect, but of course this is a "band-aid" solution again. In this respect, I am wondering if there is a way to keep the connect 'active' with a 'poke' or something. If you guys have any knowledge on maintaining active websocket connections with users, let me know.

I definitely need to do some research on AWS for this issues, but I think some knowledge on the ParlAI websocket program would also be very helpful, as I currently do not know it very well. I'll likely study the code later, but if you guys, the ParlAI devs, could provide a brief overview of the websocket program for the chatbot service, that would really help me get a better understanding of the program overall.

[stephenroller]

Hmm, do you have any monitor of RAM?

[jakeane]

Not over a period of time, but I just set up a cronjob to log the memory every 30 minutes. Currently, with 8gb of RAM and after 100 minutes of runtime, 2gb is used, 2gb is free, 3.7gb is buff/cache, and 5.5gb is available. We will see how these numbers change over the course of a few hours.

EDIT: So a few hours ago the chatbot restarted, and in result, 1gb of ram got freed. So, 1gb is now being used, and 3gb is free. I think we can establish that the websocket eats up RAM overtime, but what's the cause? It may be dormant conversations, as I do not ever send "[DONE]" to the websocket.

[stephenroller]

Hmm okay, ya I expect something maybe isn't freeing up on a timeout. I'm not too familiar with the chat services code, but I'll ask @klshuster for ideas this week and see if I can look at it.

[github-actions[bot]]

This issue has not had activity in 30 days. Please feel free to reopen if you have more issues. You may apply the "never-stale" tag to prevent this from happening.