Mastodon: Streaming notification broken in a new installation

Created on 4 Apr 2017 · 22Comments · Source: tootsuite/mastodon

Hi,

I followed the manual installation instructions on Debian stretch, and it worked \o/

However, notifications do not seem to work, and I get a lot of errors in the logs of the mastodon-streaming service:

Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435 Error: Missing access token
Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435     at authenticationMiddleware (/home/mastodon/live/streaming/index.js:122:17)
Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435     at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)
Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435     at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:312:13)
Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435     at /home/mastodon/live/node_modules/express/lib/router/index.js:280:7
Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435     at Function.process_params (/home/mastodon/live/node_modules/express/lib/router/index.js:330:12)
Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435     at next (/home/mastodon/live/node_modules/express/lib/router/index.js:271:10)
Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435     at allowCrossDomain (/home/mastodon/live/streaming/index.js:77:3)
Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435     at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)
Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435     at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:312:13)
Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435     at /home/mastodon/live/node_modules/express/lib/router/index.js:280:7
Apr 04 01:44:13 mastodon npm[28961]: ERR! 2e2dc530-3da0-4e98-949b-d19a49bb2435  { [Error: Missing access token] statusCode: 401 }

Looking at the nginx logs, these errors are caused by a client accessing the API, but it passes an access_token:

::ffff:XX:XX:XX:XX - - [04/Apr/2017:01:44:13 +0200] "GET //api/v1/streaming/?access_token=YYYYYYYYYYY&stream=user HTTP/1.1" 401 50 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36"

Also, there are access tokens in the database (oauth_access_tokens), and the last one matches the token sent by the client.

By the way, I am running nginx + nginx + mastodon (so, two reverse proxies, one in front of the other), maybe this causes some issues?

I am currently running 4c53af64f0b10bc11473df5e3fd1cd7a11b755f6.

bug

Source

zorun

👍4

Most helpful comment

Oh my, it actually got copied when I copypasted the text from the mail. Good job.

Now Apache loads correctly and the streaming works ! And I don't need nginx anymore !

That was very helpful, thank you lots @zoski !

AxelTerizaki on 7 Apr 2017

😄2

All 22 comments

If that may help, I'm getting the same error with Apache 2 (without nginx and with one reverse proxy)

Is the double // normal on the GET url ?

I updated a few minutes ago and the error is still there.

zoski on 5 Apr 2017

Same thing on my instance, 401 errors just like @zorun's in _journalctl -u mastodon-streaming.service_, initially 404 errors in my Apache log, then 401's after a quick config fix.

Komic on 5 Apr 2017

I am using the herkou deployment from latest at master, same issue. Nothing in the timeline and logs show 404's on //api/v1/streaming/ requests.

robjloranger on 5 Apr 2017

It seems that the streaming code expects an Authorization HTTP header. In my tests, the javascript frontend does not send this header, which chauses the 401 error.

zorun on 5 Apr 2017

apparently you need to follow someone from another instance to get things in federated, and more users.

robjloranger on 5 Apr 2017

So, the problem was that nginx was not correctly setup to proxy websocket connections.

Just adding proxy_set_header Connection "upgrade"; to the configuration block for the API solved the issue, see: https://nginx.org/en/docs/http/websocket.html

zorun on 6 Apr 2017

Duh. Thanks, I'll give a shot to https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html on my side.

Komic on 6 Apr 2017

For nginx, this was actually in the installation documentation (docs/Running-Mastodon/Production-guide.md), I just read it badly...

@Komic If you manage to get it to work with Apache, contribute to the doc!

zorun on 6 Apr 2017

Will do.

Komic on 6 Apr 2017

@zoski @komic, I am using the following successfully with Docker Mastodon, behind Apache

   ProxyPreserveHost On
   ProxyRequests Off
   RequestHeader set X_FORWARDED_PROTO 'https'

   ProxyPass /api/v1/streaming ws://localhost:4000
   ProxyPassReverse /api/v1/streaming ws://localhost:4000

   ProxyPass / http://localhost:3000/
   ProxyPassReverse / http://localhost:3000/

I also have LOCAL_HTTPS=true in my .env.production .

mcg on 6 Apr 2017

👍2

Yup @mcg, tried a few minutes ago, it's now working perfectly.
(do you really need the ProxyPassReverse for the API too?)

Komic on 6 Apr 2017

@Komic I assume so, as you'd want to mangle headers coming out of that proxy.

mcg on 6 Apr 2017

Hi @Komic @mcg thank you for the heads up !

It's now working for me. I forget to enable proxy_wstunnel in the first place but it's fine now.

zoski on 6 Apr 2017

👍1

I seem to be running into the same issue, except I'm running apache + nginx + mastodon.
I'd actually love to find a fully working apache config instead of nginx since it'd allow me to get rid of it (I'm not using it for anything else)

For now here is my nginx conf :

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

server {
  listen 6443 ssl;
  server_name shelter.moe;

  ssl_certificate     /etc/ssl/certs/ssl-cert-snakeoil.pem;
  ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
  ssl_protocols TLSv1.2;
  ssl_ciphers EECDH+AES:EECDH+CHACHA20;
  ssl_prefer_server_ciphers on;

  keepalive_timeout    70;
  sendfile             on;
  client_max_body_size 0;
  gzip off;

  root /home/mastodon/live/public;

  add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload";

  location / {
    try_files $uri @proxy;
  }

  location @proxy {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;

    proxy_pass_header Server;

    proxy_pass http://127.0.0.1:3000;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    tcp_nodelay on;
  }
 location /api/v1/streaming {
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto https;
  proxy_pass http://127.0.0.1:4000;
  proxy_buffering off;
  proxy_redirect off;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;
  tcp_nodelay on;
 }

  error_page 500 501 502 503 504 /500.html;
}

And my apache one :

<VirtualHost *:443>
        ServerName shelter.moe
        DocumentRoot /home/www/shelter.moe
        ProxyPreserveHost On
        ProxyRequests On
        <Directory /home/www/shelter.moe>
                require all granted
        </Directory>
        <Location />
                ProxyPass https://localhost:6443/
                ProxyPassReverse https://localhost:6443/
                SSLRequireSSL
        </Location>
        SSLEngine on
        SSLProxyEngine On
        SSLProxyVerify none
        SSLProxyCheckPeerCN Off
        SSLProxyCheckPeerName off
        SSLProxyCheckPeerExpire off
        SSLOptions +StdEnvVars
        SSLCertificateKeyFile /etc/letsencrypt/live/shelter.moe/privkey.pem
        SSLCertificateFile /etc/letsencrypt/live/shelter.moe/cert.pem
        SSLCertificateChainFile  /etc/letsencrypt/live/shelter.moe/fullchain.pem
        Include /etc/apache2/sites-available/vhosts_logging.include
        ErrorLog /home/www/shelter.moe/logs/error_log
        TransferLog /home/www/shelter.moe/logs/access_log
</VirtualHost>

I get these kind of errors in the streaming service :

ERR! 36df7473-27e6-400c-8a34-4b7f5e53960c     at authenticationMiddleware (/home/mastodon/live/streaming/index.js:122:17)
ERR! 36df7473-27e6-400c-8a34-4b7f5e53960c     at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)
ERR! 36df7473-27e6-400c-8a34-4b7f5e53960c     at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:312:13)
ERR! 36df7473-27e6-400c-8a34-4b7f5e53960c     at /home/mastodon/live/node_modules/express/lib/router/index.js:280:7
ERR! 36df7473-27e6-400c-8a34-4b7f5e53960c     at Function.process_params (/home/mastodon/live/node_modules/express/lib/router/index.js:330:12)
ERR! 36df7473-27e6-400c-8a34-4b7f5e53960c     at next (/home/mastodon/live/node_modules/express/lib/router/index.js:271:10)
ERR! 36df7473-27e6-400c-8a34-4b7f5e53960c     at allowCrossDomain (/home/mastodon/live/streaming/index.js:77:3)
ERR! 36df7473-27e6-400c-8a34-4b7f5e53960c     at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)
ERR! 36df7473-27e6-400c-8a34-4b7f5e53960c     at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:312:13)
ERR! 36df7473-27e6-400c-8a34-4b7f5e53960c     at /home/mastodon/live/node_modules/express/lib/router/index.js:280:7
ERR! 36df7473-27e6-400c-8a34-4b7f5e53960c  { [Error: Missing access token] statusCode: 401 }
```` 

I tried switching to

proxy_set_header Connection "upgrade";
```
In the nginx config but it didn't show any results.

If anyone knows what could be going wrong with this...

AxelTerizaki on 7 Apr 2017

Here is my complete Apache Vhost, I'm running without nginx as all my website are served by apache. I'm open to ameliorations on this vhost by the way.

Don't forget to do a a2enmod proxy_wstunnel otherwise the ws:// won't work. Hope this could help you. If you need content of /etc/letsencrypt/options-ssl-apache.confask for it.

<IfModule mod_ssl.c>
<Virtualhost *:443>
        ServerName masto.raildecake.fr
        ServerAdmin [email protected]

        DocumentRoot /home/mastodon/live/public
        ServerAlias masto.raildecake.fr

    <Directory "/home/mastodon/live/public">    
        Require all granted
    </Directory>

        # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
        LogLevel error ssl:warn

        ErrorLog ${APACHE_LOG_DIR}/masto.raildecake.error.log
        CustomLog ${APACHE_LOG_DIR}/masto.raildecake.log combined

    SSLCertificateFile /etc/letsencrypt/live/masto.raildecake.fr/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/masto.raildecake.fr/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf

    ProxyPreserveHost On
    ProxyRequests Off
        RequestHeader set X-Forwarded-Proto "https"

        # Proxy to Mastodon services
        ProxyPass "/api/v1/streaming" "ws://127.0.0.1:4000"
        ProxyPassReverse "/api/v1/streaming" "ws://127.0.0.1:4000"

        ProxyPass / http://127.0.0.1:3000/
        ProxyPassReverse / http://127.0.0.1:3000/

    # Custom error pages
    ErrorDocument 500 /500.html;
    ErrorDocument 501 /500.html;
    ErrorDocument 502 /500.html;
        ErrorDocument 503 /500.html;
        ErrorDocument 504 /500.html;

</Virtualhost>
</IfModule>

zoski on 7 Apr 2017

Thanks for the help !

I enabled proxy_wstunnel via a2enmod, it says it's been enabled and all, but when I restart apache, it won't load due to configtest failure :
ProxyPass Unable to parse URL: *ws*://127.0.0.1:4000

Which leaves me a little puzzled...
If I use my old config and restart apache2, I can check that the mod_proxy_wstunnel is loaded thanks to server-info.

However it doesn't look like it understands the ws syntax...

AxelTerizaki on 7 Apr 2017

Which Apache version are you running, @AxelTerizaki?

Komic on 7 Apr 2017

Weird, I may missing something. Is there the "stars" in your url ?

*ws*://127.0.0.1:4000

I'm running Apache/2.4.18 with Ubuntu 16.04

zoski on 7 Apr 2017

Oh my, it actually got copied when I copypasted the text from the mail. Good job.

Now Apache loads correctly and the streaming works ! And I don't need nginx anymore !

That was very helpful, thank you lots @zoski !

AxelTerizaki on 7 Apr 2017