Ingress-nginx: Ingress controller not upgrading WebSockets

[juselius]

NGINX Ingress controller version: 0.9.0-beta.15
Kubernetes version (use kubectl version): v1.7.9+7f63532e4ff4f
Environment:

• Cloud provider or hardware configuration: Private/VMWare
• OS: NixOS 18.03pre118381.4068703502
• Kernel: 4.9.58
• Install tools:
• Others: helm 2.6.1, tiller 2.6.1

What happened:
I have a http service running on a pod with IP 10.1.2.3, port 80. It serves websocket connections on /ws. I can connect to it directly on the cluster:

$ curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: test.it" -H "Origin: http://ping.it" -H "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" http://10.1.2.3/ws

HTTP/1.1 101 Switching Protocols
Server: Suave (https://suave.io)
Date: Fri, 27 Oct 2017 11:20:40 GMT
Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=
Connection: Upgrade
Upgrade: websocket
Content-Type: text/html

�helo?

If I try to access the service via the Ingress:

$ curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: test.it" -H "Origin: http://ping.it" -H "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" http://10.10.10.10/ws

HTTP/1.1 400 Bad Request
Server: nginx/1.13.5
Date: Fri, 27 Oct 2017 11:25:23 GMT
Content-Type: text/html
Content-Length: 11
Connection: keep-alive

Bad Request⏎  

This seems to suggest that the connection upgrade request is lost on the way.

My ingress manifest:

kind: Ingress
metadata:
  name: wintering-pig-test
  labels:
    app: test
    chart: test-0.1.0
    release: wintering-pig
    heritage: Tiller
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ngress.kubernetes.io/ssl-redirect: "false"
    nginx.org/websocket-services: "wintering-pig-test"
spec:
  rules:
    - host: test.it
      http:
        paths:
          - path: /
            backend:
              serviceName: wintering-pig-test
              servicePort: 80

I have examined nginx.conf on the ingress controllers, and cannot find anything out of place. In particular the config contains all the websocket related stanzas.

The ingress controller itself is installed using helm with the following overrides in values.yaml:

controller:
  hostNetwork: true
  ingressClass: nginx
  kind: Deployment
  replicaCount: 2
  service:
    type: NodePort
  nodeSelector:
    ingressController: "true"

What you expected to happen:
A response from the service.

How to reproduce it:

Anything else we need to know:

[aledbf]

@juselius please use http://www.websocket.org/echo.html as client. I cannot reproduce this issue

[aledbf]

@juselius here is an example

echo "
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ws-example
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: wseg
    spec:
      containers:
      - name: websocketexample
        image: nicksardo/websocketexample
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 8080
        env:
        - name: podname
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
---
apiVersion: v1
kind: Service
metadata:
  name: ws-example-svc
  labels:
    app: wseg
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: wseg
---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ws-example-svc
spec:
  rules:
  - host: websocket.uswest2-01.rocket-science.io
    http:
      paths:
      - backend:
          serviceName: ws-example-svc
          servicePort: 80
" | kubectl create -f -

Using http://www.websocket.org/echo.html set ws://websocket.uswest2-01.rocket-science.io as test Location

[juselius]

Thank you. I can connect to the WS echo service both via the ingress and directly on the node port using curl:

port=80
# port=32123 
curl -i -N \
    -H "Connection: Upgrade"\
    -H "Upgrade: websocket"\
    -H "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ=="\
    -H "Sec-WebSocket-Version: 13"\
    -H "Origin: http://foo.com/"\
    -H "Host: ws.cluster.local" http://ws.cluster.local:$port/ws

The Sec-WebSocket-Version: 13 header is necessary, otherwise I get a 400 Bad Request. When I connect directly (bypassing the ingress) to my own actual service, this header is not necessary.

Using the Kaazing echo service I get the following error, regardless of whether I go through the ingress or not:

VM1115:161 WebSocket connection to 'ws://ws.cluster.local/ws?.kl=Y' failed: Error during WebSocket handshake: Sent non-empty 'Sec-WebSocket-Protocol' header but no response was received

The Kaazing request reaches the WS echo server, which logs the request:

2017/10/30 09:30:27 Received request 10.10.59.0:39012
2017/10/30 09:30:27 Error while reading: websocket: close 1006 (abnormal closure): unexpected EOF

Any help pinpointing where the error could be is much appreciated!

[aledbf]

I am sorry but I'm not familiar with kaazing. Maybe you should open an issue in the kaazing repository?

[aledbf]

@juselius can we close this issue?

[juselius]

@aledbf Sorry, forgot to mention that http://www.websocket.org/echo.html redirects to http://demos.kaazing.com/echo/.

If nobody has any idea what could be wrong, I guess we can close the issue. But the underlying problem is still unresolved.

[aledbf]

But the underlying problem is still unresolved.

I am sorry but unless you post a complete deployment with the service you are trying to use there is nothing we can do to help.
This is the code of the example that I used to show that websockets works ootb:
https://github.com/kubernetes/ingress-gce/tree/master/examples/websocket

[juselius]

I think this issue can be closed (for now). I changed the manifests to use the nicksardo/websocketexample image instead of the actual service, and everything seems to work correctly. I guess I just have to roll up the sleeves, fire up wireshark, and start debugging using the raw traffic... Thank you for your help.

[juselius]

I found the bug using WireShark. The problem is in the Suave.IO web server library used by the endpoint service. Suave does not ignore the case of the http headers, and when the ingress rewrites Connection: Upgrade to Connection: upgrade, Suave responds with HTTP/1.1 400 Bad request.

[aledbf]

@juselius can you use a custom template and replace upgrade with Upgrade?

[juselius]

@aledbf yes, that's the quick fix. I'm currently preparing a PR for Suave, before this bites other people. Thanks again for all the help!

[akhilputhiry]

I am facing the same issue. getting 400

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ws-example
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: wseg
    spec:
      containers:
      - name: websocketexample
        image: nicksardo/websocketexample
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 8080
        env:
        - name: podname
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
---
apiVersion: v1
kind: Service
metadata:
  name: ws-example-svc
  labels:
    app: wseg
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: wseg
---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ws-example-svc
spec:
  rules:
  - host: ws.dev.lti-mosaic.com
    http:
      paths:
      - backend:
          serviceName: ws-example-svc
          servicePort: 80

[prakasa-tkpd]

need heelp, what happen with my configuration, I can test to direct backend and everything is fine. This is what I get from ingress nginx and direct from the pods. Using kubernetes bare metal kubeadm in ubuntu 18 and private VPS.

Ingress:

curl --include \
     --no-buffer \
     --header "Connection: Upgrade" \
     --header "Upgrade: websocket" \
     --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
     --header "Sec-WebSocket-Version: 13" \
     https://www.terpusat.com/graphql
HTTP/2 400
date: Thu, 07 May 2020 15:34:27 GMT
content-type: text/plain; charset=utf-8
content-length: 18
x-xss-protection: 1; mode=block
x-dns-prefetch-control: off
x-frame-options: SAMEORIGIN
x-download-options: noopen
x-content-type-options: nosniff
vary: Origin

GET query missing.%

Pod:

curl --include \
     --no-buffer \
     --header "Connection: Upgrade" \
     --header "Upgrade: websocket" \
     --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
     --header "Sec-WebSocket-Version: 13" \
     http://localhost:3000/graphql
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=

My ingress configuration is very basic:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
  name: nginx-ingress
spec:
  rules:
  - host: www.terpusat.com
    http:
      paths:
      - path: /
        backend:
          serviceName: anakin
          servicePort: 3000

any other configuration I missed ?, need help...

I didn't my configuration from pod. Still using this in any block servers

proxy_set_header                        Connection        $connection_upgrade;