Kong: Error during WebSocket handshake: 'Upgrade' header is missing

[ourfor]

• Kong 2.0.2
• Server Config

{
    "host": "api.dev.ourfor.top",
    "created_at": 1584945910,
    "connect_timeout": 60000,
    "id": "46f07b22-d0e8-4f5e-a05f-eb3f57e4da54",
    "protocol": "http",
    "name": "api",
    "read_timeout": 60000,
    "port": 80,
    "path": null,
    "updated_at": 1584945910,
    "retries": 5,
    "write_timeout": 60000,
    "tags": null,
    "client_certificate": null
}

• Route Config

{
    "id": "de596b50-c3b9-48aa-98e2-5d3d7e245230",
    "path_handling": "v0",
    "paths": null,
    "destinations": null,
    "headers": null,
    "protocols": [
        "http",
        "https"
    ],
    "methods": null,
    "snis": null,
    "service": {
        "id": "46f07b22-d0e8-4f5e-a05f-eb3f57e4da54"
    },
    "name": null,
    "strip_path": true,
    "preserve_host": false,
    "regex_priority": 0,
    "updated_at": 1584945983,
    "sources": null,
    "hosts": [
        "demo.ourfor.top"
    ],
    "https_redirect_status_code": 426,
    "tags": null,
    "created_at": 1584945983
}

http works ok, but can't connect websocket, what should I do?

[bungle]

I think it is your upstream that does not handle the request correctly. Kong will send:
Connection: keep-alive, Upgrade, and your upstream websocket implementation just cannot parse that, aka it thinks there is no Upgrade in connection while there is? Am I correct? And Kong also sends Upgrade: websocket. We do have a test for this here:

https://github.com/Kong/kong/blob/master/spec/02-integration/05-proxy/03-upstream_headers_spec.lua#L166-L180

Can you check on your upstream what headers the request contains?

[ourfor]

I use tomcat and the log show, the onError execute

2020-03-23 08:00:15.046 ERROR 1 --- [nio-8080-exec-3] k.r.j.i.KClassImpl                       : LoginSocket: 出现异常
2020-03-23 08:00:15.046 ERROR 1 --- [nio-8080-exec-3] k.r.j.i.KClassImpl                       : java.io.EOFException

any good idea?

[bungle]

@ourfor,

If you change this line:
https://github.com/Kong/kong/blob/2.0.2/kong/runloop/handler.lua#L1120

var.upstream_connection = "keep-alive, Upgrade"

TO

var.upstream_connection = "Upgrade"

Does it work then?

[ourfor]

I use docker image 😅

[ourfor]

build next branch source code and change

tomcat log

it seems like connection closed early

[bungle]

In docker image the files should be at:

/usr/local/share/lua/5.1/kong

But the issue is rather strange. Is there anything between:

1. client and kong?
2. kong and upstream?

[ourfor]

I change the file in docker image, got this error:

WebSocket connection to 'ws://demo.ourfor.top:8000/auth/wechat' failed: Error during WebSocket handshake: Unexpected response code: 404

tomcat log

23-Mar-2020 10:56:49.110 INFO [http-nio-8080-exec-8] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
 Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
    java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol
        at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:567)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:502)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1623)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

before using kong, I had used apache and iis. work well with apache, same error with iis

[ourfor]

I use nodejs to test websocket.

• server code

const Server = require('ws').Server
ws = new Server({port: 80})
ws.on('connection', ws => {
    console.log('client connected')

    ws.on('message', msg => {
        console.log(`client -> server: ${msg}`)
        ws.send('hello, I am server')
    })
})

deploy: api.dev.ourfor.topTest

• client code

const socket = new WebSocket('ws://api.dev.ourfor.top')

socket.onopen = () => {
    console.log('connected!')
    socket.send('hello, I am client.')
}

socket.onmessage = ({data: msg}) => {
    console.log(`server -> client: ${msg}`)
    console.log(msg)
}

before using kong,

after using those config:

• api service:

{
    "host": "api.dev.ourfor.top",
    "created_at": 1584966116,
    "connect_timeout": 60000,
    "id": "ffe00cc0-40ab-4109-843c-c9818efc0038",
    "protocol": "http",
    "name": "api",
    "read_timeout": 60000,
    "port": 80,
    "path": null,
    "updated_at": 1584966116,
    "retries": 5,
    "write_timeout": 60000,
    "tags": null,
    "client_certificate": null
}

• route

{
    "next": null,
    "data": [
        {
            "id": "ad8eb330-6f5e-46fe-b8b6-38d7364b537e",
            "path_handling": "v0",
            "paths": null,
            "destinations": null,
            "headers": null,
            "protocols": [
                "http",
                "https"
            ],
            "methods": null,
            "snis": null,
            "service": {
                "id": "ffe00cc0-40ab-4109-843c-c9818efc0038"
            },
            "name": null,
            "strip_path": true,
            "preserve_host": false,
            "regex_priority": 0,
            "updated_at": 1584966133,
            "sources": null,
            "hosts": [
                "demo.ourfor.top"
            ],
            "https_redirect_status_code": 426,
            "tags": null,
            "created_at": 1584966133
        }
    ]
}

and client code change to:

const socket = new WebSocket('ws://demo.ourfor.top:8000')

socket.onopen = () => {
    console.log('connected!')
    socket.send('hello, I am client.')
}

socket.onmessage = ({data: msg}) => {
    console.log(`server -> client: ${msg}`)
    console.log(msg)
}

and test http

[bungle]

@ourfor I used my machine to proxy (using Kong 2.0.2) to:
http://api.dev.ourfor.top/

Using Chrome Version 80.0.3987.149 (Official Build) (64-bit). (also tested with Firefox and works too)

[bungle]

Here is the service I used:

{
    "client_certificate": null,
    "connect_timeout": 60000,
    "created_at": 1584974390,
    "host": "api.dev.ourfor.top",
    "id": "b6838ab6-a71e-48e2-aee2-b367a8be1eac",
    "name": null,
    "path": "/",
    "port": 80,
    "protocol": "http",
    "read_timeout": 60000,
    "retries": 5,
    "tags": null,
    "updated_at": 1584974390,
    "write_timeout": 60000
}

and here is the route:

{
    "created_at": 1584974415,
    "destinations": null,
    "headers": null,
    "hosts": null,
    "https_redirect_status_code": 426,
    "id": "c669e947-a9bc-45a3-8ab1-ea83703f0f34",
    "methods": null,
    "name": null,
    "path_handling": "v0",
    "paths": [
        "/"
    ],
    "preserve_host": false,
    "protocols": [
        "http",
        "https"
    ],
    "regex_priority": 0,
    "service": {
        "id": "b6838ab6-a71e-48e2-aee2-b367a8be1eac"
    },
    "snis": null,
    "sources": null,
    "strip_path": true,
    "tags": null,
    "updated_at": 1584974415
}

[ourfor]

the problem is that I have a static website and api service in one server. I use blog.ourfor.top to visit my
static website and use api.ourfor.top to provide data. How should I configure?

api service contains http and websocket

[bungle]

Do you know why it works on my machine? I think it should work on yours as well. I don’t see why it won’t work with static website.

[ourfor]

set service path to /, it worked. thank you!

service:

curl -i -X POST \
--url http://localhost:8001/services \
--data "name=api" \
--data "url=http://api.dev.ourfor.top" \
--data "path=/"

route

curl -i -X POST \
--url http://localhost:8001/services/api/routes \
--data "paths[]=/" \
--data "hosts[]=demo.ourfor.top"

test nodejs

test tomcat

[bungle]

@ourfor, great! thanks for getting back. I am closing this as it turned out to be a small configuration issue.

[ourfor]

[UPDATE]
It seems like can't work well with tomcat

• service

{
    "host": "api.dev.ourfor.top",
    "created_at": 1585711143,
    "connect_timeout": 60000,
    "id": "ed72b4f9-8e89-4211-b86f-cec1a56841b5",
    "protocol": "http",
    "name": "api",
    "read_timeout": 60000,
    "port": 80,
    "path": "/",
    "updated_at": 1585711351,
    "retries": 5,
    "write_timeout": 60000,
    "tags": null,
    "client_certificate": null
}

• route

{
    "next": null,
    "data": [
        {
            "id": "d1287f25-4e1e-4d77-926d-1f5b0928febd",
            "path_handling": "v0",
            "paths": [
                "/"
            ],
            "destinations": null,
            "headers": null,
            "protocols": [
                "http",
                "https"
            ],
            "methods": null,
            "snis": null,
            "service": {
                "id": "ed72b4f9-8e89-4211-b86f-cec1a56841b5"
            },
            "name": null,
            "strip_path": true,
            "preserve_host": false,
            "regex_priority": 0,
            "updated_at": 1585711151,
            "sources": null,
            "hosts": [
                "demo.ourfor.top"
            ],
            "https_redirect_status_code": 426,
            "tags": null,
            "created_at": 1585711151
        }
    ]
}

• websocket

const socket1 = new WebSocket('ws://api.dev.ourfor.top/auth/wechat')
socket1.onopen = () => {
    console.log('socket1 connected')
}

const socket2 = new WebSocket('ws://demo.ourfor.top:8000/auth/wechat')
socket2.onopen = () => {
    console.log('socket2 connected')
}

result is WebSocket connection to 'ws://demo.ourfor.top:8000/auth/wechat' failed: Error during WebSocket handshake: 'Upgrade' header is missing 😂

[ourfor]

follow https://github.com/Kong/kong/issues/5714#issuecomment-602500322, it worked

[elvis824]

@bungle I also encountered the same issue with Tomcat.
With Kong v2.0.3, websocket connections from both Chrome or Firefbox would be dropped by Tomcat 9.0.29 if it is behind Kong.

Following the changes you mentioned in https://github.com/Kong/kong/issues/5714#issuecomment-602500322, or downgrading Kong to v2.0.0, everything works again.

[dwin]

Also having this issue with Java Spring, looking into resolving with https://github.com/Kong/kong/issues/5714#issuecomment-602500322

[zcpwillam]

I got the same problem, the client parsed upgrade response, and when I checked the headers of the upgrade response, there was no key named "upgrade", which caused the following validation exception;

[dwin]

The Upgrade header is still there, but it is not in the order that these servers expect and I think this is an issue on the upstream side that it doesn't inspect more than one value for each header key.

This is the Dockerfile we use to fix it, we're _not actually building Kong_, just updating a line in the handler for this header.

FROM alpine:3.12 AS helper

WORKDIR /scratch

RUN apk add --no-cache sed
# copy handler file from Kong image to modify
COPY --from=kong:2.1.0-alpine /usr/local/share/lua/5.1/kong/runloop/handler.lua .
# Apply modification to Websocket handling https://github.com/Kong/kong/issues/5714#issuecomment-602514521
RUN sed -i 's/var.upstream_connection = "keep-alive, Upgrade"/var.upstream_connection = "Upgrade"/' handler.lua

FROM kong:2.1.0
# Copy modified file in
COPY --from=helper /scratch/handler.lua /usr/local/share/lua/5.1/kong/runloop/handler.lua
# no other changes

[jasine]

same issue