Aws-load-balancer-controller: How to use websocket in ALB using aws-alb-ingress-controller

[vrathore18]

I am having a K8s setup in AWS which is working fine but due to some requirement I need to enable websocket in one of my application.

I tried everything mentioned on #189
but I am still getting the error as he mentioned.

https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/189#issuecomment-390313396

Also Aws-alb-ingress-controller creating 1 alb per ingress. Can we do something to fix this as this will increase the cost

[M00nF1sh]

@vrathore18 can you share your client/server setup? (i don't think you are using websocket but are using another websocket-like protocol like socket.io, socket.io != websocket)

For plain websocket, you don't need to do any thing special, e.g.

• You don't need to change sessionAffinity on service to be clientIP
• You don't need to enable stickiness on targetGroup

But, you need to make sure two things are configured properly:

• Your ingress rule accepts the initial http request. (e.g. /m00nf1sh if you are connecting with ws://<alb_dns>/m00nf1sh)
• You do heartbeats(ping/pong) to keep the connections active. You can adjust the idle timeout with
  alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=600).

If you are using other protocols, you need to make sure the HTTP requests can be handled and are being handled by same server.
For socket.io, it initially doing client-side long polling to simulate websocket like behavior before try to upgrade to use an transport protocol like websocket.

• It will send requests like GET /socket.io/?transport=polling&EIO=3&t=1575497560.840519 HTTP/1.1, so you need to ensure your Ingress handles such request path. (e.g. /socket.io/ instead of just /, wildcards like /* works fine too), Note: the path is configurable on your socket.io server/client, while defaults to 'socket.io'
• You need configure the heartbeats properly(pingInterval). so that the heartbeats keeps the
  connection(idle_timeout.timeout_seconds) and stickiness warm(stickiness.lb_cookie.duration_seconds).
• You need to enable stickiness on target group to ensure the requests are been handled by same backend: alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60
• You need to use mode IP using alb.ingress.kubernetes.io/target-type: ip to ensure the requests are been handled by same backend. (the default instance target-type won't work)

[vrathore18]

@M00nF1sh : Thanks for helping.
I am using https://github.com/apollographql/subscriptions-transport-ws websocket for my application. So my current setup: aws-alb-ingress-controller + ingress. And for each ingress one alb is getting created(thats a different issue).

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: {{ template "pd.frontend.fullname" . }}
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: {{ .Values.ingressScheme }}
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
    alb.ingress.kubernetes.io/certificate-arn: {{ .Values.external.acmCertificateARN }}
    alb.ingress.kubernetes.io/inbound-cidrs: {{ .Values.ipWhiteList }}
spec:
  rules:
    - host: app.{{ .Values.domainName }}
      http:
        paths:
          - path: /*
            backend:
              serviceName: {{ template "pd.frontend.fullname" . }}
              servicePort: 80

Do I have to update the configuration on subscriptions-transport-ws as well to make it working?

[M00nF1sh]

@vrathore18
I'm haven't used this project,
would you help try below to check whether it works:

1. add alb.ingress.kubernetes.io/target-type: ip to your Ingress
2. set keepAlive option as 30000 (30sec) to https://github.com/apollographql/subscriptions-transport-ws/blob/master/src/server.ts#L91

[vrathore18]

@M00nF1sh : thanks I try that out. Also I don't need to enable stickiness?

[M00nF1sh]

@vrathore18 u need to enable stickness as well

[vrathore18]

Hey @M00nF1sh . I have tried everything you suggested but still websocket is not working. Below is how I am deploying the sample app. Could you please check.

use case: Prisma + Mysql and testing the alb for websockets.

Mysql-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: database
  namespace: default
spec:
  ports:
  - port: 3306
    targetPort: 3306
    protocol: TCP
  selector:
    stage: production
    name: database
    app: mysql

mysql pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: database-disk
  namespace: default
  labels:
    stage: production
    name: database
    app: mysql
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

mysql-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: database
  namespace: default
  labels:
    stage: production
    name: database
    app: mysql
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        stage: production
        name: database
        app: mysql
    spec:
      containers:
        - name: mysql
          image: 'mysql:5.7'
          args:
            - --ignore-db-dir=lost+found
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: "prisma"
          ports:
            - name: mysql-3306
              containerPort: 3306
          volumeMounts:
            - name: database-disk
              readOnly: false
              mountPath: /var/lib/mysql
      volumes:
        - name: database-disk
          persistentVolumeClaim:
            claimName: database-disk

prisma-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prisma-configmap
  namespace: default
  labels:
    stage: production
    name: prisma
    app: prisma
data:
  PRISMA_CONFIG: |
    port: 4466
    # uncomment the next line and provide the env var PRISMA_MANAGEMENT_API_SECRET=my-secret to activate cluster security
    # managementApiSecret: my-secret
    databases:
      default:
        connector: mysql
        host: database
        port: 3306
        user: root
        password: prisma
        migrations: true

prisma-deploy.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prisma
  namespace: default
  labels:
    stage: production
    name: prisma
    app: prisma
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        stage: production
        name: prisma
        app: prisma
    spec:
      containers:
        - name: prisma
          image: 'prismagraphql/prisma:1.34'
          ports:
            - name: prisma-4466
              containerPort: 4466
          env:
            - name: PRISMA_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: prisma-configmap
                  key: PRISMA_CONFIG

prisma-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: prisma
  namespace: default
spec:
  type: NodePort
  ports:
  - port: 4466
  selector:
    stage: production
    name: prisma
    app: prisma

And finally our ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prisma
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
    alb.ingress.kubernetes.io/certificate-arn: {{ .Values.external.acmCertificateARN }}
    alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60
    alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=600
    alb.ingress.kubernetes.io/inbound-cidrs: <my ip address>/32
spec:
  rules:
    - host: pp.{{ .Values.domainName }}
      http:
        paths:
          - path: /*
            backend:
              serviceName: prisma
              servicePort: 4466

I tried locally. The app prisma+mysql is working fine but when I am creating the ingress. Target group and alb is getting created but not able to get the websocket work with alb.

[vrathore18]

@M00nF1sh : any update on this?

[dannysid]

any update? same issue can't seem to make websocket work. It's like the connection and upgrade headers are removed somewhere in the process.

[walkafwalka]

If you upvote the original post, it will help show your support for this to be looked into.

[ceecer1]

does anyone know how to make it work?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[lucasmacosta]

I did everything suggested by @M00nF1sh on this comment and after that socket.io worked as intended, both long polling and sockets

[achisolomon]

happens to #metoo

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yspreen]

I found this issue because I was looking for a way to change the timeout. The keyword I was looking for was keepalive_timeout, because both nginx and gke ingress refer to it with that name. Maybe you can add a comment to the documentation, that alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=X can be used to change the keepalive timeout?