Cri-o: log_size_max is ignored. Logs always wrap at 10M

[ssoor]

Description

log_size_max in crio.conf is not honored. Pod logs always wrap around 10MB

CRI-O 1.11.11 on k8s 1.14.0

# kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

# crictl version
Version:  0.1.0
RuntimeName:  cri-o
RuntimeVersion:  1.11.11-1.rhaos3.11.git474f73d.el7
RuntimeApiVersion:  v1alpha1

Steps to reproduce the issue:

1. Edit /etc/crio/crio.conf and set log_size_max = -1
2. systemctl restart crio (also tried rebooting
3. Run containers that create a lot of stdout and watch the size of the pods logs under
   /var/log/pods/ - can find the symlink under /var/log/containers

Describe the results you received:
Pod logs will wrap when they reach 10MB

Describe the results you expected:

pod logs do not wrap until they reach log_size_max bytes

Additional information you deem important (e.g. issue happens only occasionally):

Output of crio --version:

crio version 1.11.11-1.rhaos3.11.git474f73d.el7

Additional environment details (AWS, VirtualBox, physical, etc.):
VMware EXSi

[mrunalp]

That is probably done by the kubelet.

[zouyee]

That is probably defined in docker.service.

[rhatdan]

@haircommander PTAL

[mrunalp]

Can you check if you are setting log size max in kubelet configuration?

 container-log-max-size:
  - 10Mi

?

[ssoor]

Sorry to reply so late.
@zouyee We didn't use docker as the runtime, we are using cri-o.

# cat /etc/crio/crio.conf | grep max
# log_size_max is the max limit for the container log size in bytes.
log_size_max = -1

@mrunalp
The kubelet runing command is:
/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock

and kubelet does not have the option to configure the log max size:

# cat /var/lib/kubelet/config.yaml | grep max
maxOpenFiles: 1000000
maxPods: 110
# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock

[haircommander]

@ssoor it seems by https://github.com/kubernetes/kubernetes/blob/a78ae4ba74a591afd46d65cb3d76cf2929e68fcb/staging/src/k8s.io/kubelet/config/v1beta1/types.go#L657 if you don't set a log size max in the kubelet it will default to 10Mi. Can you try adding a containerLogMaxSize field to your kubelet.conf and see if it fixes your problem?

[ssoor]

@haircommander Is this reason, thank you very much.