K3d: [BUG] Server Side Events not working behind Traefik ingress in k3d

[daggerok]

What did you do?

I have streaming CQRS app with one way data flow, (see deployment, service and traefik ingress manifests), deployed into k3d. Same app will works fine if I for example using NGINX ingress in docker-for-mac, minikube or KIND.

NOTE: frontend app and traefik is working as ingress as expected, only sse stream is not here...

_k8s/ingress/traefik/k3d/traefik-for-k8s-in-k3d.yaml_

# https://docs.traefik.io/v1.7/configuration/backends/rancher/
# but sse does not works...
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-traefik-k3d
  annotations:
    # kubernetes.io/ingress.class: traefik
    traefik.frontend.rule.type: PathPrefixStrip
spec:
  backend:
    serviceName: frontend-service
    servicePort: 80
  rules:
    - http:
        paths:
          - path: /backend
            backend:
              serviceName: backend-service
              servicePort: "webflux"

_k8s/frontend.yaml_

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-service
  labels:
    app: frontend-service
spec:
  replicas: 1
  template:
    metadata:
      name: frontend-service
      labels:
        app: frontend-service
    spec:
      containers:
        - name: frontend-service
          image: daggerok/frontend
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
              protocol: TCP
              name: "webflux"
          readinessProbe:
            httpGet:
              scheme: HTTP
              port: 8080
              path: /actuator/health
          livenessProbe:
            httpGet:
              scheme: HTTP
              port: 8080
              path: /actuator/health
          env:
            - name: TWEETS_SERVER_HOST
              value: "backend-service"
            - name: TWEETS_SERVER_PORT
              value: "7000"
      restartPolicy: Always
  selector:
    matchLabels:
      app: frontend-service
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  selector:
    app: frontend-service
  sessionAffinity: None
  ports:
    - port: 80
      protocol: TCP
      nodePort: 30080
      targetPort: 8080
      name: "webflux"
  type: NodePort
---

file _k8s/backend.yaml_ files is omitted, but services logging showing that all traffic between application services gets delivered.

• How was the cluster created?

  • k3d create --name k3s --api-port 6551 --publish 80:80 --workers 1 ; sleep 5s
  • export KUBECONFIG="$(k3d get-kubeconfig --name='k3s')"
  • kubectl get pods --all-namespaces -o wide -w &
  • kubectl apply -f k8s/ -f k8s/ingress/traefik/k3d/

• What did you do afterwards?

  • k3d commands?
  • docker commands?
  • OS operations (e.g. shutdown/reboot)?

Nothing, just waiting for bootstrap and testing if it works

What did you expect to happen?

I'm expecting that next my code will work correctly, like it's working locally or with NGINX-ingress:

        const eventSource = new EventSource(`/api/stream-tweets/${author}/?${query}`);
        eventSource.onopen = () => /*... */;
        eventSource.onerror = cause => dispatch('error', { /* ... */ });
        eventSource.onmessage = message => {
          const jsonString = message.data;
          const tweet = JSON.parse(jsonString);
          commit('addMessage', tweet);
        };
        commit('connect', eventSource);

Which OS & Architecture?

• Windows, MacOS
• amd64

I have tried both (all latests at the moment): macOS and Windows 10

Which version of k3d?

• output of k3d --version

k3d version v1.4.0

Which version of docker?

• output of docker version

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

PS: Is there are some way install NGINX ingress in k3d instead of Traefik?

---

Regards,
Max

[iwilltry42]

Hi there, thanks for opening this issue.
I didn't read every detail of your manifests, so maybe I missed some info there.
But can you write here, what issues you're actually facing?
Do you get any error messages, is the ingress working at all (e.g. via curl)?

You can choose not to install traefik when creating the cluster by adding --server-arg '--no-deploy=traefik' to the k3d create command. Then you can install whatever ingress controller you like :+1:

[daggerok]

Hello, @iwilltry42

No issues at all, http-event-streaming (SSE) just not starting on frontend side if I'm using embedded traefik in standard k3d cluster installation... while other functionality (http-post and http-get rest calls) works correct over traefik ingress

Thank you for feedback, I will try --server-arg '--no-deploy=traefik' and install nginx-ingress and come back with results later

---

Regards

[daggerok]

Hello!

Thanks for help, @iwilltry42, I was able to create cluster with server args you point me and with nginx ingress controller everything is working fine.

So I can confirm that described problem was related either to k3d+traefik or just traefik (I don't know exactly)..

I will left that issue as opened, because seems like described problem is bug unless somebody will confirm that it's not related to k3d.

---

Cheers

[iwilltry42]

Hi @daggerok, glad that it works now for you!
Can you please post the ingress manifest that you use with nginx here?

[daggerok]

Hello!

Sure:

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-default-backend
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "12h"
spec:
  backend:
    serviceName: frontend-service
    servicePort: webflux
  rules:
    - http:
        paths:
          - path: /
            backend:
              serviceName: frontend-service
              servicePort: webflux
---
# that one is not necessary, just for debugging purposes:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-rewrite-target
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/proxy-read-timeout: "12h"
spec:
  rules:
    - http:
        paths:
          - path: /backend(/|$)(.*)
            backend:
              serviceName: backend-service
              servicePort: webflux
---

PS: please take a note, that prerequisites are already installed nginx ingress controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.27.0/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.27.0/deploy/static/provider/cloud-generic.yaml

like described in official installation guide

---

Regards

[iwilltry42]

Maybe this is related?
-> https://github.com/containous/traefik/issues/2576

[daggerok]

Hi,

Yeah... people saying that setting compress = false can help, but I don't know how I can do that with k3d...

[daggerok]

Found this: https://docs.traefik.io/middlewares/compress/

[iwilltry42]

@daggerok I think I can help you out there.
First of: k3s is currently coming with traefik 1.7.x by default, so you'd be looking for this part of the docs: https://docs.traefik.io/v1.7/configuration/entrypoints/#compression .
To disable the compression you can edit the configuration of traefik directly in the running k3s/k3d cluster: kubectl edit configmap -n kube-system traefik.
The default configmap contains this config:

traefik.toml: |
    # traefik.toml
    logLevel = "info"
    defaultEntryPoints = ["http","https"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      compress = true # <-- I guess you have to change this according to your comment
      [entryPoints.https]
      address = ":443"
      compress = true
        [entryPoints.https.tls]
          [[entryPoints.https.tls.certificates]]
          CertFile = "/ssl/tls.crt"
          KeyFile = "/ssl/tls.key"
    [ping]
    entryPoint = "http"
    [kubernetes]
      [kubernetes.ingressEndpoint]
      publishedService = "kube-system/traefik"
    [traefikLog]
      format = "json"
    [metrics]
      [metrics.prometheus]
        entryPoint = "traefik"

After your change, traefik should restart and pickup the changes :+1:

[daggerok]

Hello, @iwilltry42
Thanks for feedback!
I will try that and come back with updates

---

Regards

[daggerok]

Hello,

After your change, traefik should restart and pickup the changes 👍

Unfoturnately not, I found that after change you have to kill all existing traefik pods manually and only after that, deployments will be re-provisioned with new configuration in place:

kubectl delete pods -n kube-system `kubectl get pods -n kube-system | grep traefik | awk '{print $1}'`

After these manipulations my streaming application is fully working as excepted.

With all this, looks like easiest way is use --server-arg with NGINX-ingress instead.
Seems like from this point of time, that particular issues is blocking by traefik itself.
I will add our findings in Traefik project issue Server Side Events [SSE] problem with Traefik Kubernetes Ingress Controller on k3d #6161

Thank you @iwilltry42 for your support!

---

Regards,
Maksim

[iwilltry42]

Hi @daggerok

Unfoturnately not, I found that after change you have to kill all existing traefik pods manually and only after that, deployments will be re-provisioned with new configuration in place

You're right, sorry. I was mislead by the annotation on the traefik deployment: Annotations: checksum/config: 5072e2b9911158c04b7b76568f0c4f477d59793d52d6a44d35a4910b5e15527c but obviously this used by the Helm Chart to update the deployment once the configmap is updated :roll_eyes:

Anyway, I'm glad that this is working now.

Since this is not an issue with k3d or k3s and is fairly easy to work around, I'll go ahead and close this issue :+1:

[daggerok]

JFY: Traefik guys proposing use v2 with excludedContentTypes option

Do you know if k3d is planning use Traefik v2 and will it be possible to use Middleware with such option?
Thanks

[iwilltry42]

Do you know if k3d is planning use Traefik v2 and will it be possible to use Middleware with such option?

The automatically deployed traefik is coming from k3s itself, so your best bet would be to ask in the #k3s slack channel or to open an issue on https://github.com/rancher/k3s , since I'm not part of Rancher. But personally I don't see a huge problem in switching to v2 (just did it seamlessly for a personal project).
Regarding the Middleware: I have no clue, but neither k3d nor k3s should be blocking you there then.

Another tip: you can use auto-deploying manifests (see docs/issues) to automatically deploy an ingress controller of your choice (and anything else) upon cluster creation.