K3s: Traefik 2.0 integration

[Zikoel]

Is your feature request related to a problem? Please describe.
The feature tls-passthrough is missing.
Es. Installing argocd on the cluster is difficult due the missing of this feature.

Describe the solution you'd like
Substitute the actual version < 2.0 with the actual (that reach the GA with version 2.0)

Describe alternatives you've considered
Describe a reproducible way for remove the actual version in favor of the most updated version.

Additional context
The version of Traefik 2.0 seems most kubernetes friendly so, this seems to me a very natural step to do!

[sebastien-prudhomme]

It seems that Traefik 2.0 drops Ingress support and so drops compatibility with most of the community Helm charts.

So i don't think it's a good idea if it's really the case, even if Ingress has some limitations.

[Zikoel]

From the documentation Traefik & Kubernetes seems that the Ingress support is not dropped but fully supported, more of this there is another custom Ingress the with enhanced capabilities for complex routes and middlewares!

[sebastien-prudhomme]

Thanks for reading the full doc! It was not clear as also talk on migrating to their new CRD.

As k3s is using a chart to deploy Traefik, i guess we need to wait for the new version of the chart which is still not ready for production: https://github.com/containous/traefik/issues/5440

[denysvitali]

I'm using this and it works pretty well once you disable the original traefik:

write_files:
- encoding: ""
  content: |-
    apiVersion: helm.cattle.io/v1
    kind: HelmChart
    metadata:
      name: traefik
      namespace: kube-system
    spec:
      chart: traefik
      repo: https://nanosapp.github.io/chart-releases/stable
      targetNamespace: kube-system
      set:
        image.tag: 2.0.4
        ports.web.port: 80
        ports.websecure.port: 443
  owner: root
  path: /var/lib/rancher/k3s/server/manifests/traefik.yaml
  permissions: '0600'

k3os:
  dns_nameservers:
    - 1.1.1.1
    - 8.8.8.8
  labels:
    region: uk1
  k3s_args:
    - server
    - "--no-deploy=traefik"

[sotiris84]

Looking forward as well the update to Traefik 2.0

[brandond]

@denysvitali that repo appears to be gone; do you have a substitute that you're still using?

[denysvitali]

@brandond Nope, that repo should still work, look at https://github.com/nanosapp/chart-releases/tree/gh-pages/stable/traefik and you'll see the chart source. I've created that repo and the company I work for owns it 👍

[brandond]

I'll give it another try. I'm just getting started with Helm and don't quite grok the repo format; I'll admit to not actually loading the chart but just hit the repo URL from the CR with my browser and got a 404.

[denysvitali]

That's just because Chart repos do not need an index.html. If you visit the repo URL with /index.yaml appended, you'll see all of the charts available in that specific repository. Check it out! 😊

[mickkael]

The Helm chart is now production ready
https://github.com/containous/traefik-helm-chart/pull/104

https://github.com/containous/traefik-helm-chart/blob/master/traefik/Chart.yaml

[r1cebank]

Been trying to get the Helm chart to work, but had not a lot of luck, followed the documentation on Traefik's website and got it working with k3s. The ansible role is here https://github.com/r1cebank/geck/tree/traefik-v2/roles/traefik

[mamiu]

@r1cebank
Create a yaml file with following content:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: traefik
  repo: https://containous.github.io/traefik-helm-chart
  set:
    image.tag: "2.2"

Then place this file in /var/lib/rancher/k3s/server/manifests and install or run the k3s server with --disable traefik.

[torpinouche]

I am not sure no-deploy is the right one, it is listed as deprecated in the documentation here

--disable value sounds the option you should use.

[mamiu]

@torpinouche You're absolutely right! Thanks, I've adapted my comment.

[SayakMukhopadhyay]

Hi,
I see that this issue has been added to the 1.20 backlog, does that mean that Traefik 2.0 integration is only coming with kubernetes 1.20 and wont be there in 1.18.x and 1.19.x?

[notsureifkevin]

o/ greetings from Containous. If you need any assistance with the transition to v2 of Traefik please don't hesitate to reach out to myself or anyone on the team. cc @SantoDE

[kbrowder]

@notsureifkevin fwict values content is pretty different in the contanious chart, specifically values that go into the toml in the traefik v1 chart aren't there (basically you need to bring your own toml). Am I understanding that right?

[notsureifkevin]

@kbrowder I believe your observation is correct, we are working to abstract some of these use cases into more helm-friendly values.yml approach, but for the purpose of k3s, either providing a toml or specifying the command line arguments through https://github.com/containous/traefik-helm-chart/blob/master/traefik/values.yaml#L118-L129 may be the best approach at the moment. cc @SantoDE

[Thalooka]

would love to see this coming sooner since traefik 2.0 is released since over a year now

[samirsss]

Bump on this one. We've run into an issue with the current installation of k3s (bundled traefik helm chart 1.81.0), only allows single external IP. The newer versions of Traefik charts support externalIPs as an array.

[SantoDE]

If you wanna tackle that @samirsss, I guess feel free.

We are happy to help :)

[mamiu]

We use Traefik 2.3 on a k3s multi node cluster in production for a few weeks now and it's working without any issues so far.

Fist I installed the k3s cluster with the --disable=traefik option. Here's a simplified version of our install command:

curl -sfL https://get.k3s.io | bash -s - \
  --write-kubeconfig-mode "0644" \
  --disable=traefik

Then I apply a file called traefik2.yaml (for whatever reason traefik.yaml didn't work - maybe because we disabled it in the first step and then k3s automatically removes traefik.yaml files or so). We use Traefik also for the automated certificate handling process but if you want to use other solutions like the cert-manager just delete all the certificate related lines. Since we have our domains on OVH the following example has ovh as a dns challange provider, but of course you can use one of all the other providers.

apiVersion: v1
kind: Namespace
metadata:
  name: traefik

---
apiVersion: v1
kind: Secret
metadata:
  name: ovh-credentials
  namespace: traefik
type: Opaque
data:
  OVH_APPLICATION_KEY: PoBhNa...     # the base64 encoded application key
  OVH_APPLICATION_SECRET: ReFGac...  # the base64 encoded application secret
  OVH_CONSUMER_KEY: YlaEW...         # the base64 encoded consumer key
  OVH_ENDPOINT: b3ZoLWV1

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  repo: https://containous.github.io/traefik-helm-chart
  chart: traefik
  bootstrap: true
  targetNamespace: traefik
  valuesContent: |-

    additionalArguments:
      - "--certificatesresolvers.letsencrypt.acme.email=[e-mail]"
      - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
      # Let's Encrypt staging API:
      - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      # Let's Encrypt production API:
      # - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
      - "--certificatesResolvers.letsencrypt.acme.dnschallenge=true"
      - "--certificatesResolvers.letsencrypt.acme.dnschallenge.provider=ovh"

      # If you want to use jaeger tracing for traefik uncomment these lines and adjust them to your jaeger installation
      # - "--tracing=true"
      # - "--tracing.serviceName=traefik"
      # - "--tracing.jaeger=true"
      # - "--tracing.jaeger.propagation=jaeger"
      # - "--tracing.jaeger.traceContextHeaderName=uber-trace-id"
      # - "--tracing.jaeger.samplingServerURL=http://jaeger-agent.jaeger.svc.cluster.local:5778/sampling"
      # - "--tracing.jaeger.localAgentHostPort=jaeger-agent.jaeger.svc.cluster.local:6831"
      # - "--tracing.jaeger.collector.endpoint=http://jaeger-collector.jaeger.svc.cluster.local:14268/api/traces?format=jaeger.thrift"

      # If you want to use prometheus monitoring for traefik uncomment these lines
      # - "--metrics=true"
      # - "--metrics.prometheus=true"
      # - "--metrics.prometheus.addServicesLabels=true"

      - "--accesslog=true"
      - "--log.level=INFO"

    persistence:
      enabled: true
      path: /data

    podAnnotations: 
      prometheus.io/port: '8082'
      prometheus.io/scrape: 'true'

    # We will apply our own ingress rule for the dashboard in the next step
    ingressRoute:
      dashboard:
        enabled: false

    envFrom:
      - secretRef:
          name: ovh-credentials

If you don't want to use Traefiks dashboard just ignore the next step.
Otherwise make sure that the Traefik installation is done and was successful. As soon as that's the case apply a file with the following content (adjusted to your needs):

apiVersion: v1
kind: Secret
metadata:
  name: admin-dashboard-auth
  namespace: traefik
data:
  # This is the base64 and openssl encoded version of "user" as user and "password" as password
  users: dXNlcjokYXByMSRxWVFuaS9FZSRqRHNNN3pnb1RoL01QeHJZTjlxVlEvCg==

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: admin-auth
  namespace: traefik
spec:
  basicAuth:
    secret: admin-dashboard-auth

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: traefik
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`traefik.example.com`)
      services:
        - name: api@internal
          kind: TraefikService
      middlewares:
        - name: admin-auth
  tls:
    certResolver: letsencrypt
    domains:
      - main: "*.example.com"

Now Traefik 2.3 is installed in a k3s cluster with a dashboard reachable on traefik.example.com. If it doesn't work as expected please let me know because this is a highly simplified version of the one we're using and maybe I deleted something essential.

[ams0]

FWIW, I also used Traefik 2 with k3s/k3d (deployed with Helm with this script) as I wrote this article and it works just fine.

[Tautcius]

We use Traefik 2.3 on a k3s multi node cluster in production for a few weeks now and it's working without any issues so far.

Fist I installed the k3s cluster with the --disable=traefik option. Here's a simplified version of our install command:

curl -sfL https://get.k3s.io | bash -s - \
  --write-kubeconfig-mode "0644" \
  --disable=traefik

Then I apply a file called traefik2.yaml (for whatever reason traefik.yaml didn't work - maybe because we disabled it in the first step and then k3s automatically removes traefik.yaml files or so). We use Traefik also for the automated certificate handling process but if you want to use other solutions like the cert-manager just delete all the certificate related lines. Since we have our domains on OVH the following example has ovh as a dns challange provider, but of course you can use one of all the other providers.

apiVersion: v1
kind: Namespace
metadata:
  name: traefik

---
apiVersion: v1
kind: Secret
metadata:
  name: ovh-credentials
  namespace: traefik
type: Opaque
data:
  OVH_APPLICATION_KEY: PoBhNa...     # the base64 encoded application key
  OVH_APPLICATION_SECRET: ReFGac...  # the base64 encoded application secret
  OVH_CONSUMER_KEY: YlaEW...         # the base64 encoded consumer key
  OVH_ENDPOINT: b3ZoLWV1

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  repo: https://containous.github.io/traefik-helm-chart
  chart: traefik
  bootstrap: true
  targetNamespace: traefik
  valuesContent: |-

    additionalArguments:
      - "--certificatesresolvers.letsencrypt.acme.email=[e-mail]"
      - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
      # Let's Encrypt staging API:
      - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      # Let's Encrypt production API:
      # - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
      - "--certificatesResolvers.letsencrypt.acme.dnschallenge=true"
      - "--certificatesResolvers.letsencrypt.acme.dnschallenge.provider=ovh"

      # If you want to use jaeger tracing for traefik uncomment these lines and adjust them to your jaeger installation
      # - "--tracing=true"
      # - "--tracing.serviceName=traefik"
      # - "--tracing.jaeger=true"
      # - "--tracing.jaeger.propagation=jaeger"
      # - "--tracing.jaeger.traceContextHeaderName=uber-trace-id"
      # - "--tracing.jaeger.samplingServerURL=http://jaeger-agent.jaeger.svc.cluster.local:5778/sampling"
      # - "--tracing.jaeger.localAgentHostPort=jaeger-agent.jaeger.svc.cluster.local:6831"
      # - "--tracing.jaeger.collector.endpoint=http://jaeger-collector.jaeger.svc.cluster.local:14268/api/traces?format=jaeger.thrift"

      # If you want to use prometheus monitoring for traefik uncomment these lines
      # - "--metrics=true"
      # - "--metrics.prometheus=true"
      # - "--metrics.prometheus.addServicesLabels=true"

      - "--accesslog=true"
      - "--log.level=INFO"

    persistence:
      enabled: true
      path: /data

    podAnnotations: 
      prometheus.io/port: '8082'
      prometheus.io/scrape: 'true'

    # We will apply our own ingress rule for the dashboard in the next step
    ingressRoute:
      dashboard:
        enabled: false

    envFrom:
      - secretRef:
          name: ovh-credentials

If you don't want to use Traefiks dashboard just ignore the next step.
Otherwise make sure that the Traefik installation is done and was successful. As soon as that's the case apply a file with the following content (adjusted to your needs):

apiVersion: v1
kind: Secret
metadata:
  name: admin-dashboard-auth
  namespace: traefik
data:
  # This is the base64 and openssl encoded version of "user" as user and "password" as password
  users: dXNlcjokYXByMSRxWVFuaS9FZSRqRHNNN3pnb1RoL01QeHJZTjlxVlEvCg==

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: admin-auth
  namespace: traefik
spec:
  basicAuth:
    secret: admin-dashboard-auth

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: traefik
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`traefik.example.com`)
      services:
        - name: api@internal
          kind: TraefikService
      middlewares:
        - name: admin-auth
  tls:
    certResolver: letsencrypt
    domains:
      - main: "*.example.com"

Now Traefik 2.3 is installed in a k3s cluster with a dashboard reachable on _traefik.example.com_. If it doesn't work as expected please let me know because this is a highly simplified version of the one we're using and maybe I deleted something essential.

not working for me...

[lexfrei]

@Tautcius you can try this. Warks perfectly for me (with metallb, but you can exclude it)
https://github.com/lexfrei/k8s/blob/master/helmfile.d/traefik.yaml

[Tautcius]

@mamiu is it K3S cluster?

[mamiu]

@Tautcius

We use Traefik 2.3 on a k3s multi node cluster in production for a few weeks now and it's working without any issues so far.

The first sentence in my comment above. So, yes it is.

But we really should relocate this discussion because there are 20 participants in this issue (the number is already decreasing), and everyone gets an email for each comment. I think the Rancher Forum for k3s, k3OS and k3d might be the right place.

[cjellick]

@innobead I'd like to see this move forward. We'll discuss it later today. If you or your team picked up, I'd want to start with a "spike" investigation effort where the engineer investigates it and writes up their findings/design.

My biggest, number 1 question is: how will it impact upgrades? Do upgraded clusters just stay on traefik 1? That is probably the easiest way forward. Assuming thats the case, would there be a way to "opt in" to the traefik 2.

The other question I have is around CRD management. Reading above, it seems like the traefik 2 integration intoduces a new CRD. Managing CRD defintions through helm charts has been known to cause some problems. So, I'd want to make sure we are covering our bases there.

[SantoDE]

Im also here to help :)

[cjellick]

Thanks @SantoDE!

Any thoughts or commments on my above concerns?

[SantoDE]

Yes. Will provide my feedback tommorow @cjellick

[SantoDE]

Hey all,

For Upgrading path: Imho, upgrading clusters by default should stay on Traefik 1, based on a changed configuration. It can cause troubles with Traefik 2 if there is no change done upfront. Therefore, if possible, Id try to make it an explicit opt-in. Im not sure about k3s here tough :)

CRDs are always a pain to manage to be honest. We ship the most current versions with our helm chart of course, but thats as we need to it. Its not super convienient either. I could imagine something different here in the future, but for now, its what we have. It would probably require to bump the CRDs here as well, as soon as we have a new version of Traefik which requires a changed CRD, no?

cc: @cjellick

[darkcloud784]

We use Traefik 2.3 on a k3s multi node cluster in production for a few weeks now and it's working without any issues so far.

Fist I installed the k3s cluster with the --disable=traefik option. Here's a simplified version of our install command:

curl -sfL https://get.k3s.io | bash -s - \
  --write-kubeconfig-mode "0644" \
  --disable=traefik

Then I apply a file called traefik2.yaml (for whatever reason traefik.yaml didn't work - maybe because we disabled it in the first step and then k3s automatically removes traefik.yaml files or so). We use Traefik also for the automated certificate handling process but if you want to use other solutions like the cert-manager just delete all the certificate related lines. Since we have our domains on OVH the following example has ovh as a dns challange provider, but of course you can use one of all the other providers.

apiVersion: v1
kind: Namespace
metadata:
  name: traefik

---
apiVersion: v1
kind: Secret
metadata:
  name: ovh-credentials
  namespace: traefik
type: Opaque
data:
  OVH_APPLICATION_KEY: PoBhNa...     # the base64 encoded application key
  OVH_APPLICATION_SECRET: ReFGac...  # the base64 encoded application secret
  OVH_CONSUMER_KEY: YlaEW...         # the base64 encoded consumer key
  OVH_ENDPOINT: b3ZoLWV1

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  repo: https://containous.github.io/traefik-helm-chart
  chart: traefik
  bootstrap: true
  targetNamespace: traefik
  valuesContent: |-

    additionalArguments:
      - "--certificatesresolvers.letsencrypt.acme.email=[e-mail]"
      - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
      # Let's Encrypt staging API:
      - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      # Let's Encrypt production API:
      # - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
      - "--certificatesResolvers.letsencrypt.acme.dnschallenge=true"
      - "--certificatesResolvers.letsencrypt.acme.dnschallenge.provider=ovh"

      # If you want to use jaeger tracing for traefik uncomment these lines and adjust them to your jaeger installation
      # - "--tracing=true"
      # - "--tracing.serviceName=traefik"
      # - "--tracing.jaeger=true"
      # - "--tracing.jaeger.propagation=jaeger"
      # - "--tracing.jaeger.traceContextHeaderName=uber-trace-id"
      # - "--tracing.jaeger.samplingServerURL=http://jaeger-agent.jaeger.svc.cluster.local:5778/sampling"
      # - "--tracing.jaeger.localAgentHostPort=jaeger-agent.jaeger.svc.cluster.local:6831"
      # - "--tracing.jaeger.collector.endpoint=http://jaeger-collector.jaeger.svc.cluster.local:14268/api/traces?format=jaeger.thrift"

      # If you want to use prometheus monitoring for traefik uncomment these lines
      # - "--metrics=true"
      # - "--metrics.prometheus=true"
      # - "--metrics.prometheus.addServicesLabels=true"

      - "--accesslog=true"
      - "--log.level=INFO"

    persistence:
      enabled: true
      path: /data

    podAnnotations: 
      prometheus.io/port: '8082'
      prometheus.io/scrape: 'true'

    # We will apply our own ingress rule for the dashboard in the next step
    ingressRoute:
      dashboard:
        enabled: false

    envFrom:
      - secretRef:
          name: ovh-credentials

If you don't want to use Traefiks dashboard just ignore the next step.
Otherwise make sure that the Traefik installation is done and was successful. As soon as that's the case apply a file with the following content (adjusted to your needs):

apiVersion: v1
kind: Secret
metadata:
  name: admin-dashboard-auth
  namespace: traefik
data:
  # This is the base64 and openssl encoded version of "user" as user and "password" as password
  users: dXNlcjokYXByMSRxWVFuaS9FZSRqRHNNN3pnb1RoL01QeHJZTjlxVlEvCg==

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: admin-auth
  namespace: traefik
spec:
  basicAuth:
    secret: admin-dashboard-auth

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: traefik
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`traefik.example.com`)
      services:
        - name: api@internal
          kind: TraefikService
      middlewares:
        - name: admin-auth
  tls:
    certResolver: letsencrypt
    domains:
      - main: "*.example.com"

Now Traefik 2.3 is installed in a k3s cluster with a dashboard reachable on _traefik.example.com_. If it doesn't work as expected please let me know because this is a highly simplified version of the one we're using and maybe I deleted something essential.

Mine seems to be stuck on pending when attempting to deploy this. I have a 2 master, 3 worker node cluster.
my pod describe comes back with the following.

Warning FailedScheduling 10m default-scheduler 0/5 nodes are available: 2 node(s) had taint {CriticalAddonsOnly: true}, that the pod didn't tolerate, 3 node(s) didn't match node selector.
Warning FailedScheduling 10m default-scheduler 0/5 nodes are available: 2 node(s) had taint {CriticalAddonsOnly: true}, that the pod didn't tolerate, 3 node(s) didn't match node selector.

[brandond]

It's telling you right there in the error what tolerations or node selectors you need to add to allow it to schedule.

[FlexibleToast]

This issue is over a year old now and I see some movement is getting done recently. That's awesome to see. Another reason to lump on to the original is that Kubernetes has deprecated networking.k8s.io/v1beta1 ingress in 1.14 and is removing it entirely in 1.22. This is easily visible by querying about ingress on the cluster

$ kubectl get ingress
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
No resources found in default namespace.

Due to this it looks as though Traefik is adding networking.k8s.io/v1 support in their 2.4 release. Hopefully upgrading Traefik in k3s means it can be upgraded to the latest 2.x version.

[innobead]

cc @c3y1huang please help with this.

[brandond]

This is going to be a hard one to support upgrades to since traefik 2 does things completely differently. It might need to be an opt-in.

[innobead]

This is going to be a hard one to support upgrades to since traefik 2 does things completely differently. It might need to be an opt-in.

Yes, this was what we discussed today as well as per the migration note ref: https://doc.traefik.io/traefik/migration/v1-to-v2/, but @c3y1huang will still spend some time to figure out the complete context first.

[SantoDE]

This issue is over a year old now and I see some movement is getting done recently. That's awesome to see. Another reason to lump on to the original is that Kubernetes has deprecated networking.k8s.io/v1beta1 ingress in 1.14 and is removing it entirely in 1.22. This is easily visible by querying about ingress on the cluster

$ kubectl get ingress
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
No resources found in default namespace.

Due to this it looks as though Traefik is adding networking.k8s.io/v1 support in their 2.4 release. Hopefully upgrading Traefik in k3s means it can be upgraded to the latest 2.x version.

Traefik v2 already supports networking.k8s.io/v1 as the K8s API is doing some mapping. The merge of the PR to upgrade the ingress handling will be delayed for 2.5, as this will put a base k8s version requirement which we don't want to introduce on such a short notice.

For you @c3y1huang, if you need some more information / help please reach out. Im / we are more than happy to help here moving things forward.