Amazon-eks-ami: Does Docker do log rotation?

[cantino]

According to https://kubernetes.io/docs/concepts/cluster-administration/logging/, log rotation for Docker is required to not fill up the disk. GCE has it by default if kube-up.sh is used, but I can't find anything rotating /var/log/*.log in this repository. Here's how it works for gce.

Am I correct that if I run Docker images on EKS for long enough, it will crash with full disks?

[micahhausler]

You're correct, we don't have any application level log rotation on by default. We'll take this as a feature request

[cantino]

Thanks @micahhausler. Is there a recommended path forward in the mean time? Should we connect to the EC2 instances and install logrotate for a particular path?

[micahhausler]

Yes, you can update your logrotate configuration on boot, or build your own AMI

[dpiddockcmp]

The logrotate linked (non-expiring link here) does not rotate container logs: it rotates logs for various other k8s services.

Docker itself needs to be configured to rotate logs as per this support article. Which can be seen here in the GCE kube-up.sh script.

[max-rocket-internet]

We get some pods evicted due to disk pressure. Could this be why? Should I make a PR to add something similar to the configuration from @dpiddockcmp link to install-worker.sh?

[nickdgriffin]

This is exactly what happened to us too. The Engine is installed with vanilla configuration, so there was no container log rotation (which can be set with log-opts) and it filled the disk. It also doesn't pick up /etc/docker/daemon.json automatically, so having to update the service definition to feed in "--config-file=/etc/docker/daemon.json".

[bryantbiggs]

After looking into this issue, I don't think there is an optimal solution at the instance level. See my PR and closing response here, but it looks like using the image spec args section might be the best place to handle this solution

[zknill]

To clarify, is it recommended that we setup _both_ logrotate for /var/log/*.log and docker engine log rotation? Or would one of the two suffice (i.e. just logrotate)?

[nickdgriffin]

I couldn't see anything for /var/log/*.log that would need rotating - kube-proxy already has rotation enabled.

I'm going to be adding log rotation to daemon.json and configuring the daemon to read it so that container logs are rotated.

[dpiddockcmp]

This is exactly what happened to us too. The Engine is installed with vanilla configuration, so there was no container log rotation (which can be set with log-opts) and it filled the disk. It also doesn't pick up /etc/docker/daemon.json automatically, so having to update the service definition to feed in "--config-file=/etc/docker/daemon.json".

I have this in my user-data before calling /etc/eks/bootstrap.sh and it's working as expected:
echo '{ "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "10" }}' > /etc/docker/daemon.json && systemctl restart docker

# docker inspect 7ee33186340f | jq '.[0].HostConfig.LogConfig'`
{
  "Type": "json-file",
  "Config": {
    "max-file": "10",
    "max-size": "10m"
  }
}
# ls /var/lib/docker/containers/7ee33186340f*
7ee33186340f2c2a3598e5b0f76e064a42e690bf86c23e05c324dd1684fa257f-json.log
7ee33186340f2c2a3598e5b0f76e064a42e690bf86c23e05c324dd1684fa257f-json.log.1
7ee33186340f2c2a3598e5b0f76e064a42e690bf86c23e05c324dd1684fa257f-json.log.2
7ee33186340f2c2a3598e5b0f76e064a42e690bf86c23e05c324dd1684fa257f-json.log.3
7ee33186340f2c2a3598e5b0f76e064a42e690bf86c23e05c324dd1684fa257f-json.log.4
7ee33186340f2c2a3598e5b0f76e064a42e690bf86c23e05c324dd1684fa257f-json.log.5
7ee33186340f2c2a3598e5b0f76e064a42e690bf86c23e05c324dd1684fa257f-json.log.6
7ee33186340f2c2a3598e5b0f76e064a42e690bf86c23e05c324dd1684fa257f-json.log.7
7ee33186340f2c2a3598e5b0f76e064a42e690bf86c23e05c324dd1684fa257f-json.log.8
7ee33186340f2c2a3598e5b0f76e064a42e690bf86c23e05c324dd1684fa257f-json.log.9
checkpoints/
config.v2.json
hostconfig.json

I'm using amazon-eks-node-v24 (ami-0440e4f6b9713faf6) in us-east-1 created by terraform-aws-eks module

[max-rocket-internet]

@dpiddockcmp make a PR 😁 (to this repo)

[max-rocket-internet]

How about this? https://github.com/awslabs/amazon-eks-ami/pull/74

[micahhausler]

Fixed with #74

[jesseshieh]

Thanks everyone who helped with this issue.

My node-problem-detector.log file grew to several gigabytes which caused disk pressure on my node today. Is there a recommended way to deal with that file since the npd pod writes to the file mounted as a volume instead of writing to stdout? My guess is that #74 only helps with the latter case so something like logrotated is needed for files like this.

I also noticed that the journal was growing very large so I vacuumed it. Does it make sense to "rotate" the journal as well? For example, by setting SystemMaxUse.

[max-rocket-internet]

Is there a recommended way to deal with that file since the npd pod writes to the file mounted as a volume instead of writing to stdout?

NPD, and any container, shouldn't be logging directly to the file system.

[whereisaaron]

@jesseshieh you can configure you app or container build to log to stdout/stderr, 12factor.net style, and let k8s log and rotate the logs. This could be as simple as adding a soft link for the current log file to /dev/stdout.

[jesseshieh]

Ah sorry, I thought npd was installed and managed by EKS and behaved this way by default, but it looks like it is something we installed. I'll look into changing the way the log is written.

What do you think about the journal size though?

[fazil1987]

@jesseshieh you can configure you app or container build to log to stdout/stderr, 12factor.net style, and let k8s log and rotate the logs. This could be as simple as adding a soft link for the current log file to /dev/stdout.

just one confirmation : what you are saying is " if all my containers logs to stdout, k8 will handle the log rotation at node level " is that right ?

Also its mentioned here https://kubernetes.io/docs/concepts/cluster-administration/logging/ that k8 default logging doesn't support multiline logs ? please confirm

[whereisaaron]

just one confirmation : what you are saying is " if all my containers logs to stdout, k8 will handle the log rotation at node level " is that right ?

Corrected: No k8s doesn't, EKS still uses dockerd and relies on it to rotate logs.

Also its mentioned here https://kubernetes.io/docs/concepts/cluster-administration/logging/ that k8 default logging doesn't support multiline logs ? please confirm

That's correct, k8s just collects container stdout/stderr to a place where your preferred log streamer can deal with it. It is very common to use CNCF fluentd in k8s clusters, which parses the raw logs into structured JSON logs with rich k8s metadata added, like container names, pod names, pod labels, and all the good stuff to make the streamed and aggregated logs ready for ingestion, search, alerting etc. etc. You can ingest where you want, but CloudWatch is close at hand for EKS clusters.

[max-rocket-internet]

No k8s doesn't, the kube-proxy just writes the per-container log files

kube-proxy has nothing to do with logging of containers.

logrotate typically installed on k8s nodes like the EKS AMI does.

That logrotate configuration is to rotate the logs of kube-proxy itself.

just one confirmation : what you are saying is " if all my containers logs to stdout, k8 will handle the log rotation at node level " is that right ?

Correct. Technically it's not k8s, it's the docker daemon that does this, the config for that is here: https://github.com/awslabs/amazon-eks-ami/blob/master/files/docker-daemon.json#L4-L7

[fazil1987]

just one confirmation : what you are saying is " if all my containers logs to stdout, k8 will handle the log rotation at node level " is that right ?

_Corrected:_ No k8s doesn't, EKS still uses dockerd and relies on it to rotate logs.

Well, that means inside EKS, we don't have to worry about log rotation at node level. right ?

Also its mentioned here https://kubernetes.io/docs/concepts/cluster-administration/logging/ that k8 default logging doesn't support multiline logs ? please confirm

That's correct, k8s just collects container stdout/stderr to a place where your preferred log streamer can deal with it. It is very common to use CNCF fluentd in k8s clusters, which parses the raw logs into structured JSON logs with rich k8s metadata added, like container names, pod names, pod labels, and all the good stuff to make the streamed and aggregated logs ready for ingestion, search, alerting etc. etc. You can ingest where you want, but CloudWatch is close at hand for EKS clusters.

Perfect. We 're planning to use fluent-kuberenetes-daemonset, but any experience on https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions for parsing multi-line stack traces to single log message ?

[nterry]

Theres a log rotate helm chart here: https://github.com/stakater-charts/logrotate

[aneeskA]

just one confirmation : what you are saying is " if all my containers logs to stdout, k8 will handle the log rotation at node level " is that right ?

_Corrected:_ No k8s doesn't, EKS still uses dockerd and relies on it to rotate logs.

Well, that means inside EKS, we don't have to worry about log rotation at node level. right ?

@max-rocket-internet
Is this correct? I am observing pod eviction in my aws eks cluster with The node was low on resource: ephemeral-storage. I was under the impression that log rotation was on by default. k8s version is 1.15, platform version is eks.2 and AMI ID amazon-eks-node-1.15-v20200507 (ami-0c1bd9eca9c869a0d). What am I missing?

[Gabber]

Hi guys I'm running in the same issue with ami: amazon-eks-node-1.18-v20210112 (ami-0dd0589ee7a07c236), am I missing a configuration or something?