Mattermost-server: Failed to upgrade websocket connection

[demedos]

Summary

After setting up a reverse proxy the log is full of those messages:

[2017/02/13 08:53:13 PST] [EROR] websocket connect err: websocket: could not find connection header with token 'upgrade'
[2017/02/13 08:53:13 PST] [EROR] /api/v3/users/websocket:connect code=500 rid=czz3h9kqtjfb9jz3jo1sqmujne uid=1erjogmb7jraucpy1e1wifcmfy ip=192.168.10.1:56489 Failed to upgrade websocket connection [details: ]

Steps to reproduce

Set up mattermost and do a reverse proxy with IIS

Expected behavior

No error messages

Observed behavior

My website, _abc.company.com_ goes on a Windows Server with IIS, passes the request to an internal ubuntu server machine _192.168.x.x:8065_. Everything is fine except those ws:// error messages

_The error messages:_

_The reverse proxy on IIS:_

_The error message appearing on the website:_

#### Possible fixes

[jasonblais]

Thanks @demedos for the report,

Did you try these troubleshooting steps? https://docs.mattermost.com/install/troubleshooting.html?highlight=unreachable#please-check-connection-mattermost-unreachable-if-issue-persists-ask-administrator-to-check-websocket-port

Also, could we have your help posting this to the Troubleshooting Forum? There are more forum visitors than GitHub issue visitors, which means a potentially faster response. Please link back to your new post from this issue?

[demedos]

It seems that the problem is about IIS 7.5 not supporting WebSockets, as stated here:

https://www.iis.net/learn/get-started/whats-new-in-iis-8/iis-80-websocket-protocol-support

Unluckly the only solutionn is to upgrade, so tomorrow I'll try out a reverse proxy with a newer version and will let you know if it's solved

[demedos]

I would like to close the issue but I've had no luck so far.
I've upgraded my whole system to a Windows Server 2012 and IIS 8, with the WebSockets option installed and enabled, hoping that everything would be ok, but there's still some issue.

for testing purposes I've added only 1 rewrite rule to IIS, defined as below

Now that means that whatever is passed before "://" will be used as protocol in the request to the internal machine, so http remains http and ws remains ws, so the proxy works fine.

The logs are repeatedly reporting

[2017/03/01 06:45:24 PST] [EROR] websocket connect err: websocket: origin not allowed [2017/03/01 06:45:24 PST] [EROR] /api/v3/users/websocket:connect code=500 rid=nz5r9k6f63ba7e9hyskwb5ikie uid=1erjogmb7jraucpy1e1wifcmfy ip=192.168.10.1:55120 Failed to upgrade websocket connection [details: ]

and the Developer Console
WebSocket connection to 'ws://example.com/api/v3/users/websocket' failed: Error during WebSocket handshake: Unexpected response code: 403
(see image below)

But in the System Console from Mattermost CORS is enabled.

What could possibly be wrong? This is quite a show-stopper for me

[demedos]

Could this be related to the fact that the log shows ip=192.168.10.1:55120, which I suppose is the router address, and not the caller?

[BRH-Enrian]

This is a problem with a recent change to web socket processing in Mattermost. Specifically, https://github.com/mattermost/platform/commit/1a9891f0f5671551d28be54a99155b907480cc5c

Try reverting the change to CheckOrigin. This really should mimic what is being done for HTTP connections in the server.go file (see ServeHTTP where there is the check for Origin header).

We had the same problem recently and reverting this change fixed our web socket issues.

[demedos]

I'm a windows user, any chance you branch and compile it?

[jasonblais]

@demedos @BRH-Enrian are you using multiple Mattermost URLs through proxy forwarding?

If so, what is your SiteURL field in config.json (or in System Console > General > Configuration)?

We have a ticket to investigate a cross-origin issue introduced by 3.6.2: https://mattermost.atlassian.net/browse/PLT-5635

[demedos]

@jasonblais What do you mean by multiple Mattermost URLs? My current flow is
Externail IP -> IIS Server -> Linux Server.

My SiteURL property was set as https://website.com.
About the CORS, it's not working with the * nor with https://website.com.

It seems like this has something to do with the fact that Application Request Routing is stripping off the Request-Origin header from the request, or something like that.
http://stackoverflow.com/questions/34316825/websockets-reverse-proxy-in-iis-8

I hope this will be fixed asap as this is a showstopper for us since IIS is the only available web server.

[BRH-Enrian]

IMO, the issue is that the default behavior for web socket processing by Mattermost does not honor CORS setting in the Mattermost config. The code used to accept any connection request, but now it relies on default behavior as documented at http://www.gorillatoolkit.org/pkg/websocket http://www.gorillatoolkit.org/pkg/websocket under “Origin Considerations” which states that

If the CheckOrigin field is nil, then the Upgrader uses a safe default: fail the handshake if the Origin request header is present and not equal to the Host request header.

This is clearly not going to work if you have a cross-origin situation. The best solution would be to adopt similar processing as is done in api/context.go:

if len(utils.Cfg.ServiceSettings.AllowCorsFrom) > 0 {
origin := r.Header.Get("Origin")
if *utils.Cfg.ServiceSettings.AllowCorsFrom == "" || strings.Contains(*utils.Cfg.ServiceSettings.AllowCorsFrom, origin) {
w.Header().Set("Access-Control-Allow-Origin", origin)

    if r.Method == "OPTIONS" {
        w.Header().Set(
            "Access-Control-Allow-Methods",
            strings.Join(allowedMethods, ", "))

        w.Header().Set(
            "Access-Control-Allow-Headers",
            r.Header.Get("Access-Control-Request-Headers"))
    }
}

}

where instead of setting headers just return true.

Brad

On Mar 6, 2017, at 10:24, Alexander [e-mail] wrote:

@jasonblais https://github.com/jasonblais What do you mean by multiple Mattermost URLs? My current flow is
Externail IP -> IIS Server -> Linux Server.

My SiteURL property was set as https://website.com.
About the CORS, it's not working with the * nor with https://website.com.

It seems like this has something to do with the fact that Application Request Routing is stripping off the Request-Origin header from the request, or something like that.
http://stackoverflow.com/questions/34316825/websockets-reverse-proxy-in-iis-8
I hope this will be fixed asap as this is a showstopper for us since IIS is the only available web server.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/mattermost/platform/issues/5386#issuecomment-284343824, or mute the thread https://github.com/notifications/unsubscribe-auth/AX8Bpr4vPL57Vbt_Omh0TcxFGng79iKhks5ri9DrgaJpZM4L_eH6.

[demedos]

@BRH-Enrian That makes sense now, it almost drove me crazy. That means that it doesn't matter the Web server, the cross-origin requests will be refused.
Could you or @jasonblais compile me a working version please? I've already spent a whole day just trying to set up a go envoironment to compile without the edit in this commit 1a9891f

[bradhowes]

See if this works for you: https://github.com/bradhowes/platform/releases/tag/v3.6.1-websocket-cors

[demedos]

@bradhowes Thank you, but as soon I click theSave button everything in the Enable cross-origin requests from: field gets wiped out.
I just stopped the mattermost-service and replaced the content of the /opt/mattermost folder with the one you provided, supposing it's enough to make it work.

[bradhowes]

The build contains a stock config.json file which will overwrite anything you may already have there if you just blast it into /opt/mattermost. Sorry. Best bet is to untar in a tmp directory and then move the bin/platform executable into /opt/mattermost/bin. Alternatively, save the config.json file first, do what you did, and then restore the file. The build was packaged using Mattermost own Makefile.

[demedos]

That's exactly what I did. Saved the config.json, replaced all and then restored it. Anyway, changing the value manually works, the value is shown in the system console.

Now the error is changed:

WebSocket connection to 'ws://website.com/api/v3/users/websocket' failed: Error during WebSocket handshake: Incorrect 'Sec-WebSocket-Accept' header value

This is the web request sent from the browser

This may be related to the stack overflow post i mentioned before

[demedos]

@bradhowes Any way to tell it this is the correct header?

[bradhowes]

That looks OK to me. This is what is giving you an error? I’m not an expert on WebSocket protocol, but that appears normal. Note that from https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers the Sec-WebSocket-Accept value should have been calculated from the Sec-WebSocket-Key value:

The Sec-WebSocket-Accept part is interesting. The server must derive it from the Sec-WebSocket-Key that the client sent. To get it, concatenate the client's Sec-WebSocket-Key and "258EAFA5-E914-47DA-95CA-C5AB0DC85B11" together (it's a "magic string https://en.wikipedia.org/wiki/Magic_string"), take the SHA-1 hash https://en.wikipedia.org/wiki/SHA-1 of the result, and return the base64 https://en.wikipedia.org/wiki/Base64 encoding of the hash.

You could follow the above to verify that the received 'accept' value matches the given 'key' value. If not, then something is wrong in your connection flow.

Brad

On Mar 7, 2017, at 12:16, Alexander [e-mail] wrote:

@bradhowes https://github.com/bradhowes Any way to tell it this is the correct header?

https://cloud.githubusercontent.com/assets/16702156/23654215/eac8aec8-032f-11e7-8bee-1cc534c08830.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/mattermost/platform/issues/5386#issuecomment-284693728, or mute the thread https://github.com/notifications/unsubscribe-auth/AAp7YlRx13JmiEOx6uAQpJbF58g4pFBQks5rjTyrgaJpZM4L_eH6.

[demedos]

You are right, it doesn't match.

Given en81vWzPOxZRZye6CzZeZA== as the key, and concatenating it with the magic string, the key is

en81vWzPOxZRZye6CzZeZA==258EAFA5-E914-47DA-95CA-C5AB0DC85B11
which converted to sha1 hash, gives the resulting base64 string
ePXOJuIT6J656wlvecIwVSJJFmQ=

While my browser gets this Response Header:
Sec-WebSocket-Accept: YDYcdpVyxmj4L4mwi1OQA0KuvZU=.

I should test the connection from whitin the reverse proxy server I suppose.

[demedos]

@bradhowes
FWIW I'm using an Nginx web server right now and everything is just fine. This has totally to do with IIS and more specifically ARR 3, as stated here, which doesn't support the Sec-WebSocket-Extensions header.
Is there any way, or could it be implemented, to create the WebSocketServer without the perMessageDeflate attribute?
like the SO post says:
const wss = new WebSocketServer({ port: 3011, perMessageDeflate: false });
This would make Mattermost scalable to Windows Server 2012 and IIS 8.5.

[jasonblais]

@bradhowes would your PR resolve the issues described here? https://github.com/mattermost/platform/pull/5667

[BRH-Enrian]

@jasonblais Not sure. @demedos reported that something worked under Nginx, but not clear if that was with my patch or not.

[demedos]

@jasonblais The patch @bradhowes provided here worked for me, but I have to edit the config file manually because the web system console is not persisting my changes after I hit the save button, not sure if it is a normal behaviour using that patch.

Moreover, it is working using Nginx, because (at least) IIS 8.5 in Windows Server 2012, more specifically its Application Request Routing (ARR) module used to do reverse proxy, has a bug which tells the browser that he supports the per-message compression permessage-deflate, but in fact it doesn't, so it ends up giving back a wrong Sec-WebSocket-Accept key.

So yes, the patch is working, but IIS 8.5 is not supported because of its module bug (ref. here).

[attzonko]

I am seeing the exact same error/behavior with my installation. I am using Apache for the reverse proxy. One other difference is I have TLS enabled so my error is with wss://

I tried the patch provided above but the issue did not go away.

Please, let me know if I should file a new issue, where I can attach configuration and screen-captures.

[demedos]

@attzonko Have you edited the config file manually? Do you see the correct value in the System Console after editing the config.json file?

[attzonko]

@demedos I assume you mean when I tried the patch above. Yes I did edit the config.json file manually to match my setup.

Since my post I ended up installing Nginx alongside Apache, and have nginx doing the reverse proxy with SSL and the upgrade for websocket and it is indeed working. Nginx passes through all my legacy apps traffic on to Apache.

[demedos]

@attzonko Yes, I meant that. What was the exact error you had though? I'd still like to know if ARR is being buggy.

[attzonko]

That was exactly the error I had. Here is a quick cut/paste from my logs:

[2017/03/13 13:33:51 PDT] [EROR] /api/v3/users/websocket:connect code=500 rid=1raacdn1dbnmbqcuf36p6ghf6o uid=jtpx4pgkw7ny7ekeojuf8ut65r ip=10.19.142.6 Failed to upgrade websocket connection [details: ]
[2017/03/13 13:33:57 PDT] [EROR] websocket connect err: websocket: origin not allowed
[2017/03/13 13:33:57 PDT] [EROR] /api/v3/users/websocket:connect code=500   rid=nxakyx3kx7dpzpxgmbc648ms5a uid=jtpx4pgkw7ny7ekeojuf8ut65r ip=10.19.142.6 Failed to upgrade websocket connection [details: ]
[2017/03/13 13:34:01 PDT] [EROR] websocket connect err: websocket: origin not allowed
[2017/03/13 13:34:01 PDT] [EROR] /api/v3/users/websocket:connect code=500 rid=kjypmbtbbif1xqhgu3tt1zu3to uid=jtpx4pgkw7ny7ekeojuf8ut65r ip=10.19.142.6 Failed to upgrade websocket connection [details: ]
[2017/03/13 13:34:09 PDT] [EROR] websocket connect err: websocket: origin not allowed
[2017/03/13 13:34:09 PDT] [EROR] /api/v3/users/websocket:connect code=500 rid=adna1bi5hfyh38mps7rtnb1xgw uid=jtpx4pgkw7ny7ekeojuf8ut65r ip=10.19.142.6 Failed to upgrade websocket connection [details: ]
[2017/03/13 13:37:21 PDT] [EROR] websocket connect err: websocket: origin not allowed
[2017/03/13 13:37:21 PDT] [EROR] /api/v3/users/websocket:connect code=500 rid=mc1jcyf9itbzimkifbbdis96ey uid=jtpx4pgkw7ny7ekeojuf8ut65r ip=10.19.142.6 Failed to upgrade  websocket connection [details: ]
[2017/03/13 13:41:24 PDT] [EROR] websocket connect err: websocket: origin not allowed`

[attzonko]

Here is what my apache config looked like:

<VirtualHost *:443>
  ServerName mm.example.com:443
  ServerAlias mm.example.com
  CustomLog /var/log/apache2/mm_access.log combined
  ErrorLog /var/log/apache2/mm_error.log
  <Proxy *>
      Order deny,allow
      Allow from all
  </Proxy>

  SSLEngine on
  SSLProxyEngine on
  SSLCertificateFile /etc/apache2/ssl/mm_ssl.crt
  SSLCertificateKeyFile /etc/apache2/ssl/ssl.key

  RewriteEngine On
  RewriteCond %{REQUEST_URI} ^/api/v3/users/websocket [NC,OR]
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
  RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
  RewriteRule .* wss://mm.example.com:8065%{REQUEST_URI} [P,QSA,L]
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule .* https://mm.example.com:8065%{REQUEST_URI} [P,QSA,L]
  RequestHeader set X-Forwarded-Proto "https"

    <Location /api/v3/users/websocket>
      Require all granted
      ProxyPassReverse wss://mm.example.com:8065/api/v3/users/websocket
    </Location>

    <Location />
      Require all granted
      ProxyPassReverse https://mm.example.com:8065/
    </Location>

</VirtualHost>

[demedos]

I've upgraded to 3.7.1 and nothing broke, even if the websocket.go has the CheckOrigin: nil row. I suppose only Nginx is supported then. This may be related to the headers in the image below

[jasonblais]

Hi all,

PR to add websocket CORS support got merged last Thursday: https://github.com/mattermost/platform/pull/5667

This should help resolve most of the issues described here.

We've also created a PR to add the following troubleshooting steps to our install guide:

1. Upgrade to a Mattermost server v3.8.0 or later, which adds WebSocket CORS support
2. Follow the installation guide to configure NGINX as a proxy for Mattermost server: https://docs.mattermost.com/install/install-ubuntu-1604.html#configuring-nginx-as-a-proxy-for-mattermost-server
3. If you're doing reverse proxy with IIS, upgrade to IIS 8.0 and later, as WebSocket protocol is not supported in earlier versions: https://www.iis.net/learn/get-started/whats-new-in-iis-8/iis-80-websocket-protocol-support

Please let me know if there's anything else to address, otherwise we'll close the issue in a couple of days.

[demedos]

In IIS WebSockets must be enabled, since the option is not on by default. Maybe it could help in the documentation. Right now I'm using NGINX so afaik that's all

[jasonblais]

Ah okay, I'll add that note as well, doesn't hurt. Thanks @demedos!

[zwiglm]

Erm. Read it all... since I got the same error.
Have I missed something? There is no final solution for IIS atm, right?

Edit: Running Community 3.7.3 on IIS 8 with MySql (Websockets enabled and the Rewrite Rules from Unofficial Enhancements - Production Install on Windows 2012)

Edit 2: Ah, I see. A bug in IIS 8.x with ARR

[zwiglm]

In regard to:

FWIW I'm using an Nginx web server right now and everything is just fine. This has totally to do with IIS and more specifically ARR 3, as stated here, which doesn't support the Sec-WebSocket-Extensions header.
Is there any way, or could it be implemented, to create the WebSocketServer without the perMessageDeflate attribute?
like the SO post says:
const wss = new WebSocketServer({ port: 3011, perMessageDeflate: false });
This would make Mattermost scalable to Windows Server 2012 and IIS 8.5.

Can it be fixed to this in a close future version?

[zwiglm]

AWESOME. Works for me as of 3.8.2

[tstkenny]

Sorry to bring this thread up again. I am using mattermost 4.9.0 with apache 2.4. Trying to setup https proxy via apache and no luck. I followed and tried all of the solutions above. I supposed all those patches are included in 4.9.0 already.

Anyone would kindly provide the "ServiceSettings" part of the config.json for this setup please?

My goal is: access mm via https://hostname.domain.com:443 and apache proxy all traffic to http://127.0.0.1:8065 and ws://127.0.0.1:8065.

Thank you.

[lindy65]

Hi @tstkenny,

Thanks for your feedback,

As this thread is very old and was closed, may I ask you to open a new issue, stating what you have tried, which set-up documentation you have followed and Mattermost server version? Perhaps you can also include the logs which will help us to troubleshoot.

This way, it will be easier for our devs to help. Also, we don't officially support Apache but here are the unofficial docs we have - not sure whether you have looked at them?

• https://docs.mattermost.com/install/config-apache2.html?highlight=apache
• https://docs.mattermost.com/install/config-ssl-http2-apache2.html?highlight=apache

For continuity, please link in this issue to the new issue you open.

[yansern]

In a nutshell, make sure you have: IIS 8 + ARR 3 + WebSockets IIS extension.

Then here's a sample web.config settings (modify as necessary as this config is for somedomain.com/mattermost):

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name="Mattermost Inbound" stopProcessing="true">
                    <match url="(.*)" />
                    <action type="Rewrite" url="{C:1}://localhost:8065/mattermost/{R:1}" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="true">
                        <add input="{CACHE_URL}" pattern="^(.+)://" />
                        <add input="{HTTP_HOST}" pattern="^somedomain\.com$" />
                    </conditions>
                    <serverVariables>
                        <set name="HTTP_SEC_WEBSOCKET_EXTENSIONS" value="client_max_window_bits" />
                    </serverVariables>
                </rule>
            </rules>        
        </rewrite>
        <security>
            <requestFiltering allowDoubleEscaping="true" />
        </security>
    </system.webServer>
</configuration>