Phoenix: Upgrading to Phoenix 1.4 & Cowboy 2.x breaks websockets

[haslinger]

As the issue can be mitigated by downgrading to Cowboy 1.x, it is not critical or urgent at all.

Environment

• Elixir version
  Erlang/OTP 21 [erts-10.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [hipe]
Elixir 1.7.3 (compiled with Erlang/OTP 20)
• Phoenix version:
  phoenix (Hex package) (mix) - locked at 1.4.0 (phoenix) 56fe9a80
• NodeJS version:
  v6.14.4
• NPM version:
  3.10.10
• Operating system:
  Ubuntu 14.04.5 LTS

Expected behavior

After upgrade to Phoenix 1.4 and Cowboy 2.0 {:plug_cowboy, "~> 1.0"} Websockets work as before.

Actual behavior

After working through the upgrade process from phoenix 1.3 to 1.4, websockets don't work for me in production behind an nginx proxy nginx/1.4.6 any more: The browser shows me a 400 error code, while in the server log, I see

15:14:20.808 [info] CONNECT PanWeb.UserSocket
Transport: :websocket
Connect Info: %{}
Parameters: %{"token" => "LONGTOKENSTRINGHERE"
15:14:20.809 [info] Replied PanWeb.UserSocket :ok

They still do work in development, though.
I downgraded to Cowboy 1.x {:plug_cowboy, "~> 1.0"}, redeployed and everything is fine.

Current version of the codebase is open source.
I am happy to share any other information, that could help.

(and: Thanks for all your hard work :heart: )

[chrismccord]

Can you share more information about your reverse proxy setup? Any Nginx logs? As far as the phoenix logs are showing, it looks like everything on the Phoenix and Cowboy side is fine, so it's looking like an Nginx issue with cow2

[nicocaille]

Hi,

I am having the same issue, i would agree with Chris as the dev env sockets are working properly.

Here's my nginx config which works perfectly with 1.3

upstream lab_upstream {
  server 127.0.7.1:4000;
}

[...]

location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Cluster-Client-Ip $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://lab_upstream;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 3600;
    proxy_send_timeout 3600;
}

in the nginx logs, only have that kind of entries in the access log:

186.231.97.221 - - [16/Nov/2018:22:01:37 -0500] "GET /socket/websocket?token=&vsn=2.0.0 HTTP/1.1" 400 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36" "-"

or with token submitted

186.231.97.221 - - [16/Nov/2018:22:12:03 -0500] "GET /socket/websocket?token=SFMyNTY.g3QAAAACZAAEZGF0YWEFZAAGc2lnbmVkbgYAbF-nH2cB.fOuR2vyr4Wu4vBOYLs3i5gQBUaZepG94aS8uMA5Q4ZA&vsn=2.0.0 HTTP/1.1" 400 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36" "-"

and nothing showing up in the error log.

[haslinger]

Sure, here ist my nginx location config:

 location / {
    limit_req zone=webapp burst=100;
    limit_req_status 503;

    if (-f /etc/nginx/maintenance_pages/maintenance_pan_on.html) {
            return 503;
        }
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://127.0.0.1:8888;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }
 ```

The nginx log output of a failing websocket request looks like this:

94.136.7.79 - - [22/Nov/2018:08:10:46 +0100] "GET /socket/websocket?token=SFMyNTY.g3QAAAACZAAEZGF0YWEGZAAGc2lnbmVkbgYA4rpBOmcB.hWjpW6IwSp3RZEPrzmsrjw2paNO1Z5TkiPQENN2kATE&vsn=2.0.0 HTTP/1.1" 400 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"

Log output from erlang is:

08:10:46.833 [info] CONNECT PanWeb.UserSocket Transport: :websocket Connect Info: %{} Parameters: %{"token" => "SFMyNTY.g3QAAAACZAAEZGF0YWEGZAAGc2lnbmVkbgYA4rpBOmcB.hWjpW6IwSp3RZEPrzmsrjw2paNO1Z5TkiPQENN2kATE", "vsn" => "2.0.0"} │
08:10:46.834 [info] Replied PanWeb.UserSocket :ok
```

[Gazler]

Can you try with the following config option set please?

    proxy_http_version 1.1;

[haslinger]

@Gazler thank you! That's it.
So just to repeat for the folks who are running into this issue as well, set

proxy_http_version 1.1;

in your Nginx proxy config.

[guljarpd]

Can you try with the following config option set please?

    proxy_http_version 1.1;

@Gazler Thank You ! You saved my many hours. I was facing the same issue but i forgot that i am using NGINX. After reading this issue I reconfigured my NGINX and now it's working fine.

[StanBright]

Thanks mates! This was exactly what I needed. Apart from that, I was missing the following two configs that were necessary:

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";