K3os: Log rotation for k3s log

[agelwarg]

During testing, we recently discovered this bug in k3s:

https://github.com/rancher/k3s/issues/1628

As a side effect, the k3s log at /var/log/k3s-service.log is growing large very quickly. This bug is one problem, but there could be others or just normal growth to fill the filesystem over time. On a system with constrained storage, this is problematic. To protect against this, it would be nice to include log rotation in some form. Maybe it makes sense just to use syslog?

[agelwarg]

In my testing, it doesn't look like supervise-daemon supports the output_logger or error_logger settings, whereas start-stop-daemon does. I also found this issue for OpenRC: https://github.com/OpenRC/openrc/issues/341

[agelwarg]

I see there is already a logrotate config file, but crond is not started on boot. Is that by design or just a miss?

[dweomer]

I see there is already a logrotate config file, but crond is not started on boot. Is that by design or just a miss?

@agelwarg probably an intentional miss in that a boot_cmd or run_cmd can be specified to start the service (assuming it is configured correctly).

[philipsparrow]

My workaround solution to this is as follows:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: logrotate
  namespace: k3os-system
  labels:
    app: logrotate
spec:
  selector:
    matchLabels:
      app: logrotate
  template:
    metadata:
      labels:
        app: logrotate
    spec:
      automountServiceAccountToken: false
      initContainers:
      - name: truncate
        # Any very large log files should be truncated before rotation this saves disk
        # copytruncate (the default logrotate option) copies the file before truncating
        # e.g. 3GB k3s-service.log file becomes 6GB across files during the logrotate operation
        image: debian:10-slim # This is a scaled down image which contains coreutils 8.30
        command: ["truncate", "--size=<30M", "/host/var/log/k3s-service.log"]
        volumeMounts:
        - name: logs
          mountPath: /host/var/log
        securityContext:
          privileged: true
          runAsUser: 0
          capabilities:
            drop: ["ALL"]
      containers:
      - name: logrotate
        image: quay.io/honestbee/logrotate:latest
        env:
        - name: LOGROTATE_PATTERN
          value: /host/var/log/k3s-service.log
        - name: LOGROTATE_ROTATE
          value: "3" # Keep 3 files
        - name: LOGROTATE_SIZE
          value: 20M
        - name: CRON_SCHEDULE
          value: "0 5 * * *" # Every day at 05:00 UTC
        securityContext:
          # Requires write access to /etc in the pod and /var/log on the host
          privileged: true
          runAsUser: 0
          capabilities:
            drop: ["ALL"]
        resources:
          requests:
            cpu: 50m
            memory: 50Mi
          limits:
            cpu: 50m
            memory: 50Mi
        volumeMounts:
        - name: logs
          mountPath: /host/var/log
      volumes:
      - name: logs
        hostPath: # Mount the host directory
          type: Directory
          path: /var/log

[linuxmail]

hi @philipsparrow,

thanks for the workaround. My logfile was around 8GB and brought my test cluster down.