Ingress-nginx: Websockets not working on AWS w/ ingress-nginx helm package

Created on 12 Dec 2017 · 16Comments · Source: kubernetes/ingress-nginx

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: 0.9.0-beta15 (what comes with the latest stable/nginx-ingress helm package as of 2017-12-12)

Kubernetes version (use kubectl version): 1.8.5

Environment:

Cloud provider or hardware configuration: AWS
OS (e.g. from /etc/os-release): CoreOS 1576.4.0
Kernel (e.g. uname -a): 4.13.16
Install tools: kops
Others: helm

Steps to reproduce:

Create k8s cluster on AWS
1. kops create cluster --cloud=aws --state=s3://state-bucket --node-count=3 --kubernetes-version=1.8.5 --zones=us-west-2a,us-west-2b,us-west-2c --master-zones=us-west-2a,us-west-2b,us-west-2c --dns-zone=example.com --node-size=m4.2xlarge --master-size=t2.large --ssh-public-key=~/.ssh/key.pub --image=coreos.com/CoreOS-stable-1576.4.0-hvm k8s.example.com --yes
2. Wait until kops validate cluster says it's ready
Install nginx-ingress helm package
1. helm init
2. Setup basic SSL and timeout params for ingress controller
  
  echo ' controller: config: ssl-redirect: "false" proxy-read-timeout: "3600" proxy-send-timeout: "3600" replicaCount: 3 service: annotations: service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:my-acm-cert" service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600" ' > ingress-values.yaml
3. Wait until tiller is ready
4. helm install stable/nginx-ingress --values ingress-values.yaml --name my-ingress-controller
Deploy example websocket echo service
1. Use this example
  1. ...but don't forget to change the websocket.uswest2-01.rocket-science.io host param to something pointing at your AWS ELB
Browse to http://www.websocket.org/echo.html and attempt to connect to any of ws://your-elb-pointing-host.example.com/echo, ws://your-elb-pointing-host.example.com/, wss://your-elb-pointing-host.example.com/echo, or wss://your-elb-pointing-host.example.com.

Expected behavior:

The echo client would connect successfully, the Send Text button would become active, clicking it would send the Message string to the example websocket service, and the Message string would be echoed back by the example websocket service.

Observed behavior:

The echo client connects, the Send Text button remains inactive, the browser console reports a 400 error, and the connection is closed a few seconds later.
Logs from http://demos.kaazing.com/echo/:

CONNECT: ws://admin-api-k8s.turbovote.org/echo
CLOSED: (1006)
CONNECT: ws://admin-api-k8s.turbovote.org/
CLOSED: (1006)
CONNECT: wss://admin-api-k8s.turbovote.org/
CLOSED: (1006)
CONNECT: wss://admin-api-k8s.turbovote.org/echo
CLOSED: (1006)

Example log messages from nginx ingress controllers:

172.20.71.55 - [172.20.71.55] - - [12/Dec/2017:19:12:17 +0000] "GET /?.kr=xs HTTP/1.1" 400 271 "http://demos.kaazing.com/echo/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 493 0.000 [] - - - -
172.20.112.61 - [172.20.112.61] - - [12/Dec/2017:19:17:53 +0000] "GET /?.kl=Y HTTP/1.1" 400 673 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" 587 0.000 [] - - - -
172.20.37.239 - [172.20.37.239] - - [12/Dec/2017:19:17:07 +0000] "GET /echo?.kl=Y HTTP/1.1" 400 673 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" 611 0.000 [] - - - -

No output is seen in the websocket pods' logs.

Source

cap10morgan

All 16 comments

@cap10morgan you cannot terminate TLS in the ELB. You need to do that in nginx because the ELB uses a HTTP connection with nginx in that scenario

aledbf on 12 Dec 2017

This is the location of the websocket image https://github.com/kubernetes/ingress-gce/tree/master/examples/websocket

aledbf on 12 Dec 2017

@aledbf Thanks for that detail. Although it doesn't work w/o TLS either, so there's still something amiss here.

In the wss:// case I need to terminate TLS in the ELB because I'm using an ACM certificate (so I don't have access to it).

So now I'm wondering if...

The behavior I'm seeing with the ws:// protocol is still a bug.
This can turn into a feature request for supporting ELB-terminated TLS for wss:// connections.
This should be noted somewhere in the docs for ingress-nginx because this is a pretty big exception to "Support for websockets is provided by NGINX out of the box. No special configuration required."

cap10morgan on 12 Dec 2017

👍1

@cap10morgan please check the ELB is using TCP as protocol to reach the nodeport.

aledbf on 14 Dec 2017

@aledbf I confirmed it is using TCP.

cap10morgan on 14 Dec 2017

@cap10morgan

I am sorry but I cannot reproduce this issue.
This is an example using the deploy guide for AWS using the L4 option.

echo "
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ws-example
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: wseg
    spec:
      containers:
      - name: websocketexample
        image: nicksardo/websocketexample
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 8080
        env:
        - name: podname
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
---
apiVersion: v1
kind: Service
metadata:
  name: ws-example-svc
  labels:
    app: wseg
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: wseg
---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ws-example-svc
spec:
  rules:
  - host: websocket.uswest2-01.rocket-science.io
    http:
      paths:
      - backend:
          serviceName: ws-example-svc
          servicePort: 80
" | kubectl create -f -

aledbf on 14 Dec 2017

@cap10morgan the host websocket.uswest2-01.rocket-science.io will be reachable for one hour.

Edit: that example works only with ws://

aledbf on 14 Dec 2017

@cap10morgan you can use this chrome plugin https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo and use ws://websocket.uswest2-01.rocket-science.io/ws as URL

aledbf on 14 Dec 2017

@aledbf How did you provision the k8s cluster and the ingress controller? I'd like to replicate your working setup as much as possible.

cap10morgan on 14 Dec 2017

Please at least change the values S3_BUCKET_NAME, NAME

export MASTER_ZONES=us-west-2a
export WORKER_ZONES=us-west-2a,us-west-2b,us-west-2c
export KOPS_STATE_STORE=s3://k8s-uswest2-01
export AWS_DEFAULT_REGION=us-west-2

export S3_BUCKET_NAME=k8s-uswest2-01
export NAME=uswest2-01.rocket-science.io

kops create cluster \
 --name $NAME \
 --cloud aws \
 --master-zones $MASTER_ZONES \
 --zones $WORKER_ZONES \
 --master-size m4.large \
 --node-count 1 \
 --node-size m4.large \
 --ssh-public-key ~/.ssh/id_rsa.pub \
 --dns-zone $NAME  \
 --topology private  \
 --networking flannel \
 --bastion="true"  \
 --yes

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/without-rbac.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/service-l4.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/patch-configmap-l4.yaml \
    | kubectl apply -f -

aledbf on 14 Dec 2017

@cap10morgan you can just use the curl commands if you already have the cluster running

aledbf on 14 Dec 2017

Closing. Please reopen if you have more questions or issues with the procedure to deploy and test websocket connections

aledbf on 15 Dec 2017

@aledbf Why was that closed?

How can we get WebSockets working then with ELB + Nginx Ingress with SSL being terminated on ELB side?

midN on 15 Mar 2019

@midN Did you figure out how to the above? I'm having the same problems with SSL being terminated on the ELB side.

0verc1ocker on 12 Apr 2019

@0verc1ocker Nope, had to ditch websockets over https/ssl

midN on 14 Apr 2019

👀3

For others looking to solve their WebSockets/Ingress issues, I created a checklist here: https://gist.github.com/jsdevtom/7045c03c021ce46b08cb3f41db0d76da#file-ingress-service-yaml