K3s: Can't override Traefik helm manifest

[kamilgregorczyk]

Whenever I reboot master, /var/lib/rancher/k3s/server/manifests/traefik.yaml gets replaced with a new file which prevents me from making changes in it (like, setting static LB IP address)

[a01fe]

Are you running k3s server with --no-deploy traefik? It'll overwrite traefik.yaml if you don't.

[kamilgregorczyk]

Ah yes, it works for Traefik but does not for --no-deploy local-storage :/

[davidnuzik]

We are going to look into tweaking things in this area. For example, we were thinking a copy of the yaml could be created (such as naming it with a .local extension or something) and modifications made to this file would be read in upon each start of k3s. This way custom changes will not be lost. Documentation will need to be updated as well.

[BlackTurtle123]

What is the current way to keep custom changes?
I readed put them in your startup service? But this isn't documented either.

[kamilgregorczyk]

@BlackTurtle123 I changed the yaml name from Traefik.yaml to Traefik-custom.yaml, then I disabled k3s Traefik by --no-deploy Traefik and k3s did not create Traefik.yaml and it deployed my custom one :)

[BlackTurtle123]

That did the job actually

[edrex]

This is at least a documentation issue, as https://rancher.com/docs/k3s/latest/en/networking/#traefik-ingress-controller recommends editing this file to change traefik params.

[joakimr-axis]

This is at least a documentation issue, as https://rancher.com/docs/k3s/latest/en/networking/#traefik-ingress-controller recommends editing this file to change traefik params.

Update suggestion created in PR https://github.com/rancher/docs/pull/2583

[davidnuzik]

@ShylajaDevadiga as a backburner task could you look at the https://github.com/rancher/docs/pull/2583 PR and approve if you are happy with it? Would like to at least get this documentation in.

As for the improved functionality I mentioned here https://github.com/rancher/k3s/issues/1254#issuecomment-570341329 we are busy prepping for v1.19.0 and some other tasks but I have this near the top of our "next up" backlog to pull in soon. Apologies for the wait but this is still planned - just wanted to provide an update. See https://github.com/rancher/k3s/issues/1317

[ShylajaDevadiga]

@kamilgregorczyk you could also consider using HelmChartConfig
Starting from k3s version v1.19.0+k3s1, you can customize deployments through HelmChartConfig.

By default prometheus is enabled

kubectl describe pod -n kube-system traefik-7c4678d7df-rzmk4 |grep -i ports
Ports:         80/TCP, 8880/TCP, 443/TCP, 8080/TCP, 9100/TCP

In the manifest dir create a yaml example traefikconfig.yaml with below content

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    metrics:
       prometheus:
           enabled: false
    tolerations:
       - key: "db"
         operator: "Exists"
         effect: "NoSchedule"

After auto deployment

kubectl describe pod -n kube-system traefik-7b77755f5d-878c5|grep -i ports
Ports:         
80/TCP, 8880/TCP, 443/TCP

You can more details here https://rancher.com/docs/k3s/latest/en/helm/