Mastodon: Streaming not working on Heroku install

[ashfurrow]

Hi there, running into some issues with streaming not working on my instance. I'm hosting on Heroku, and I'm seeing the following 404 errors:

Apr 06 17:25:44 mastodon-technology heroku/router:  at=info method=GET path="//api/v1/streaming/?access_token=...&stream=public" host=mastodon.technology request_id=... fwd="..." dyno=web.1 connect=1ms service=20ms status=404 bytes=1566 protocol=https 
...
Apr 06 17:25:44 mastodon-technology app/web.1:  [...] method=GET path=/api/v1/streaming format=html controller=ApplicationController action=raise_not_found status=404 duration=12.25 view=0.38 db=3.92 

I took a look at the code but can't figure it out. I did notice the // at the beginning of the api call. Could that be it? It looks similar to #809 and #1016 but I'm hosting on Heroku and not Apache. Thanks for any tips :bow:

---

• [x] I searched or browsed the repo’s other issues to ensure this is not a duplicate.

[Gargron]

I am not sure. @ineffyble please help. How do we add a streaming API process (node.js) to the Heroku deployments?

[ineffyble]

I'll have a look now. That said, from my experiences, Heroku deployments utilising only free dynos aren't viable for a production instance.

[ashfurrow]

I appreciate it 🙇 If it helps, I'm using a 2x Dyno and have paid for upgrades to redis abd postgres addons. Let me know if no can provide any other details.

[ecmendenhall]

Hi all! Based on the advice from @pixeldesu over here, I was able to get the streaming API up and running on Heroku by deploying two separate apps.

As others have noted, Rails will catch requests to /api/v1/streaming/ that are meant for Node unless there's a proxy like nginx in front. Deploying the streaming API to a separate domain with shared Redis/DB creds and setting STREAMING_API_BASE_URL to the second app's config works for me. FWIW, I'm using Heroku SSL, not Cloudflare. Here's a breakdown of how things are configured between the two:

Main app:

• Custom domain and (Heroku) SSL setup.
• Postgres/Redis addons.
• Procfile: repo defaults (web runs puma, worker runs sidekiq)
• Buildpacks: repo defaults (Ruby and Node)
• Set STREAMING_API_BASE_URL to point to the second app's herokuapp.com URL.

Streaming API app:

• No custom domain, just https://nature-words-11234.herokuapp.com
• No addons.
• Procfile: just web: npm start
• Buildpacks: just heroku/nodejs. Set it explicitly with $ heroku buildpacks:set heroku/nodejs
• Set the following env vars. Heroku addons usually use full connection strings like redis://username:[email protected]:1234/dbname, so you'll have to deconstruct some of them from REDIS_URL and DB_URL:
  
  
  
  • DB_USER
  
  
  • DB_PASS
  
  
  • DB_NAME
  
  
  • DB_HOST
  
  
  • DB_PORT
  
  
  • REDIS_HOST
  
  
  • REDIS_PORT
  
  
  • REDIS_PASSWORD
  
  

[ashfurrow]

@ecmendenhall @pixeldesu thanks a lot for looking into this! I've followed the instructions above and can confirm it's working for me. One note: I had to specify STREAMING_API_BASE_URL to include the protocol https://.

Follow-up: now that the streaming is handled on another Dyno, do I still need NodeJS in my main app's build pack? I ask because I'd like to add ffmpeg to the buildpack but it puts me over the 300MB limit, and removing NodeJS would help. Is Node required for anything except the streaming server?

(Edit: I've tried removing NodeJS from the main app's buildpack and things seem okay. Will monitor.)

I'm running the streaming server on a hobby Dyno, not sure how well it will scale but I'll report back in a few days 👍

Thanks again :bow:

[ashfurrow]

Oh, one update: I did need to set PGSSLMODE to require to force SSL for the streaming Dyno, to force Heroku to connect using SSL. Hat tip to this SO question.

[Floppy]

I had a short conversation with Heroku support on this: https://twitter.com/xavriley/status/851448436473958400. I'm not sure what the solution is, but this is the documentation referred to: https://devcenter.heroku.com/articles/node-websockets#option-2-socket-io

[Floppy]

I think it means we basically need one app to proxy to the other if it gets traffic not intended for it.

[ashfurrow]

Thanks for contacting Heroku @Floppy, that's a great idea. I'm not sure how to do proxy traffic to a specific Dyno, hopefully they'll follow-up with more details. Keep us in the loop!

[Floppy]

One more tip to follow https://github.com/tootsuite/mastodon/issues/1119#issuecomment-292816340 - make the Procfile change on a separate branch (like streaming) in your mastodon fork, then you can set up heroku to autodeploy both master and streaming in your two apps.

[Floppy]

And one more. NODE_ENV=production is required in the ENV.

[Spunkie]

So as far as I understand when people say _streaming_ they are just talking about websockets and the only thing these websockets are used for is live updates to my timelines and possibly on profiles? Is that right?

Maybe I'm totally off base here but for a single user instance on a free heroku dyno wouldn't it be much cheaper and simpler for me to fallback to an ajax solution? On profiles every minute or so ajax to the current page and then replace the body of the page with the body from them the ajax request? The same thing could be done for my timelines with some additional code to ensure it does not refresh the page if I've started writing a reply/toot.

[Floppy]

Yep, I think that's right. Having it all handled by one app would certainly be simpler for Heroku deployments, but that's not the way the app is set up at the moment.

Mastodon could either (1) have an AJAX fallback, or (2) merge the websocket server into the rails server; both would mean you wouldn't need the extra server, but I'll let @gargon pass judgement on whether either would make sense or be desirable from his POV :)

[ashfurrow]

I'm going to close this issue since we've figured out the issue. I don't think falling back to AJAX requests is probably not the highest priority, but it's worth considering. @Floppy can you open a new issue?

[SerenadeX]

Looking for a way to set this up on my heroku server, following the steps doesn't make it work though. I assume a lot has changed over a year though. Are any of you still able to set up the streaming api with heroku?

[Kjwon15]

@SerenadeX Maybe uws is the problem.
I use heroku(heroku alternative) and node 10.x cannot install uws.

uws offers pre-compiled binary, if there are no compatible binary, it tries to compile. And the problem is that compilation is always failing.

If use fix node version to 8.x you might be run without problem.

[crcastle]

I got the streaming API working as a separate Heroku app. Here are all the changes I had to make to get the streaming app working on Heroku. Most of them are from the suggestions above in this thread from @ecmendenhall, @Kjwon15, @Floppy, and @ashfurrow. But I thought I'd simplify some of them and aggregate it all into one comment. This assumes you have mastodon cloned locally and a remote named heroku deployed to Heroku.

1. Create and checkout a new branch called streaming with git checkout -b streaming.
2. Change Procfile to only include web: npm start. Delete everything else.
3. Change "node": ">=6" to "node": "~8" in package.json per @Kjwon15's suggestion above to get around the uws issue. This is safe because Node 8 is in Long Term Support (being updated with security patches).
4. Add and commit these changes to your streaming branch with git add Procfile package.json and git commit -m "Changes for streaming API deploy"
5. Create a new Heroku app for the streaming API heroku create my-mastodon-streaming. Replace my-mastodon-streaming with whatever you want to call your streaming app on Heroku. App names on Heroku need to be globally unique.
6. Setup the new app as a git remote heroku git:remote --remote heroku-streaming --app my-mastodon-streaming
7. Force Heroku Postgres to use SSL with heroku config:set PGSSLMODE=require --remote heroku-streaming
8. Find the Heroku add-on attachment name for Heroku Postgres and Heroku Redis on your main Mastodon deploy with heroku addons --remote heroku. Mine were postgresql-flexible-94662 and redis-angular-97465.
9. Attach these to your new Heroku app for the streaming API with heroku addons:attach postgresql-flexible-94662 --remote heroku-streaming and heroku addons:attach redis-angular-97465 --remote heroku-streaming.
10. Explicitly set the Heroku Buildpack for the streaming app with heroku buildpacks:set heroku/nodejs --remote heroku-streaming
11. Deploy to Heroku with git push heroku-streaming streaming:master

cc @SerenadeX

[SerenadeX]

Thanks for the help but I already have moved off of Heroku and I haven't had same problems.