Cli: Docker log rotation is not working the container log keeps on growing

[avi-kalar]

I have an issue that even though I set the docker log configuration it does not kick in and my docker logs grows till I run out of disk space. Can you please tell me when the log rotation is cleaned up, I have the following config set for my container when I run

docker inspect <container-id>

      `"LogConfig": {
            "Type": "json-file",
            "Config": {
                "max-file": "5",
                "max-size": "100m"
            }
        }`

and my log size is above 400 MB right now, but the log rotation hasn't kicked in.

Steps to reproduce the issue:

1. docker run --name badguy --log-opt max-size=1k --log-opt max-file=5 -d ubuntu sh -c "while true; do printf A; done"
2. wait for a while
3. docker kill badgu

Describe the results you received:
my log size is above 1k right now, but the log rotation hasn't kicked in.

Describe the results you expected:
Log rotation with file size not exceeding 1k

Output of docker version:

Output of docker info:

Containers: 39
 Running: 1
 Paused: 0
 Stopped: 38
Images: 1441
Server Version: 17.03.1-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1214
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.13-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952 GiB
Name: moby
ID: 6GE7:EIWS:LOD2:IFS2:3UHV:A62R:FWGU:RYUM:LTSJ:7D4B:VWES:DLEL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 23
 Goroutines: 32
 System Time: 2018-06-26T10:22:08.783714835Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

[cpuguy83]

How are you checking the log size?

[kavinash-athena]

by running the command
sh -c "du -ch /var/lib/docker/containers/*/*-json.log"
and the output I get is

24K /var/lib/docker/containers/40ab2ef8b7c01f5d41d672b064fc34fcaeb51c8d2c35285e7aaf3c3f48485baa/40ab2ef8b7c01f5d41d672b064fc34fcaeb51c8d2c35285e7aaf3c3f48485baa-json.log
390M    /var/lib/docker/containers/c074baa1bd4879bee699a636ddc681884ff85b6cf5b6203cc1d5a8d75bae0e94/c074baa1bd4879bee699a636ddc681884ff85b6cf5b6203cc1d5a8d75bae0e94-json.log
8.0K    /var/lib/docker/containers/dfea013ce11098384eb4861c53ad1df3f4cf72af07fccacd5e8531f4af34e0ee/dfea013ce11098384eb4861c53ad1df3f4cf72af07fccacd5e8531f4af34e0ee-json.log
390M    total

[thaJeztah]

17.03.1 is a really old version of Docker; are you still seeing the problem if you upgrade to a current version?

[kavinash-athena]

@thaJeztah - I have an EC2 instance with the following version

docker version
Client:
 Version:   17.12.1-ce
 API version:   1.35
 Go version:    go1.9.4
 Git commit:    3dfb8343b139d6342acfd9975d7f1068b5b1c3d3
 Built: Tue Apr  3 23:37:44 2018
 OS/Arch:   linux/amd64

Server:
 Engine:
  Version:  17.12.1-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   7390fc6/17.12.1-ce
  Built:    Tue Apr  3 23:38:52 2018
  OS/Arch:  linux/amd64
  Experimental: false

Will it work on this version, and do you know how often does the job for log rotation gets triggered?

[kavinash-athena]

@thaJeztah the test container is rotating logs as expected in the above version, but I still face issues with the container I have created for my application. I am not sure why I get different behaviour in this case even though the configs are similar and the only difference is that I set the log config via ECS task

            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "5",
                    "max-size": "100m"
                }
            },

Do you see anything wrong with the config ?

The config set on ECS task definition is

      "logConfiguration": {
        "logDriver": "json-file",
        "options": {
          "max-size": "100m",
          "max-file": "5"
        }

[cpuguy83]

Is this the config that's actually set on the container in question in Docker?

[kavinash-athena]

yes @cpuguy83 this is the config that I get when run docker inspect <container-id> for the application that I wanted to have log rotation enabled. The output for list command for the container logs are

sudo ls -la /var/lib/docker/containers/63d229839904d91dc22047dc22696c4297dd7bbf7fdb50b239d6ac6b5c72624b/
total 201800
drwx------ 4 root root      4096 Jun 27 08:54 .
drwx------ 6 root root      4096 Jun 27 08:59 ..
-rw-r----- 1 root root 206595504 Jun 28 05:31 63d229839904d91dc22047dc22696c4297dd7bbf7fdb50b239d6ac6b5c72624b-json.log
drwx------ 2 root root      4096 Jun 27 08:54 checkpoints
-rw------- 1 root root      5995 Jun 27 08:54 config.v2.json
-rw-r--r-- 1 root root      1292 Jun 27 08:54 hostconfig.json
-rw-r--r-- 1 root root        13 Jun 27 08:54 hostname
-rw-r--r-- 1 root root       174 Jun 27 08:54 hosts
-rw-r--r-- 1 root root       130 Jun 27 08:54 resolv.conf
-rw-r--r-- 1 root root        71 Jun 27 08:54 resolv.conf.hash
drwxrwxrwt 2 root root        40 Jun 27 08:54 shm

The max-size configured for this container is 100m but the file has exceeded that.

[kavinash-athena]

@cpuguy83 This is working now for a smaller max-size limit, the issue was the way I have been expanding the logs to reach 100m. I was appending the same file to the end of the file and thus growing the log files. Something wasn't quite adding up. Any guess what could have gone wrong here?

[thaJeztah]

You cannot change the configuration for existing containers; when you mention "expanding the logs"; were you trying to change the configuration after containers were already started with the old configuration? Are your tools attempting to change/update the configuration of existing containers?

[cpuguy83]

Ok, so maybe some misunderstanding of these params here.
max-size is max size per file, max-file is the maximum number of files.

[wouterds]

Can confirm that it does not work as expected.

[kavinash-athena]

No, this is not working as expected, and the config is right. If you had tried to simulate this, it wouldn’t have worked though. If it is just the containers pushing logs, then the log rotation is not working as expected.

--
Thanks
Avinash

From: Wouter De Schuyter [e-mail]
Reply-To: docker/cli [e-mail]
Date: Monday, 8 October 2018 at 4:02 AM
To: docker/cli [e-mail]
Cc: Avinash Kalarikkal [e-mail], Comment [e-mail]
Subject: Re: [docker/cli] Docker log rotation is not working the container log keeps on growing (#1148)

Can confirm that it does not work as expected.

[screen shot 2018-10-08 at 00 30 45]https://user-images.githubusercontent.com/1210628/46587770-7d890300-ca91-11e8-842a-fe0660e273d8.jpg

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/docker/cli/issues/1148#issuecomment-427692854, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AVK1o2dPIcKms8TL7heGUQNsOc0u29nUks5uioDcgaJpZM4U4VeR.

[lucas-dehandschutter]

Hello,

I have a similar issue but by setting these value in the /etc/docker/daemon.json:

{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "log-level": "info",
  "iptables": true,
  "insecure-registries": [],
  "cgroup-parent": "docker",
  "icc": false
}

Even after rebooting the server and redeploying all my docker services, the logs file of each services keeps growing with no limit.

My Docker version:

Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:23 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:02:44 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[miande-acxiom]

Any update on this issue?

[widovsky]

It's not possible to change the log driver. To change it, you need to rerun the container.

[akatiyar]

I updated /etc/docker/daemon.json to {"live-restore": true, "storage-driver": "overlay2", "log-driver":"gcplogs"}, restarted the host, and I am still seeing json logs filling the disk.

[cpuguy83]

Just updating the driver in the daemon.json is not enough... as stated you must re-create the container.

[boid-com]

What is the solution to this problem? Logrotate doesn't seem to work even when containers are recreated.

[cpuguy83]

What's the exact settings in your config?
What's the container's config ("docker inspect")

[boid-com]

docker inspect HostConfig.LogConfig

            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },

docker-compose.yml

version: '3' 
services:
  prisma:
    image: prismagraphql/prisma:1.27.1
    logging:
      driver: "json-file"
      options:
          max-file: "5"
          max-size: "10m"

[cpuguy83]

This looks like compose is not passing down the options for some reason.
I'm not intimately familiar with the compose schema here.

Doing a docker run manually with log options works just fine.

[achimnol]

The log rotation is working there, but it seems not to be very accurate.
I think this may be due to I/O buffering, line-based buffering (e.g., it skips the file size check when a line output on the console is very long enough to exceed the file size), etc.

[achimnol]

We could test the log rotation by using my gist, where the first argument is the size of total output and the second argument is the length of lines.

For instance,

docker run -it \
  -v $(pwd)/long-logs.py:/root/long-logs.py \
  --log-driver json-file \
  --log-opt max-size=50 \
  --log-opt max-file=2 \
  python:3.8-alpine \
  python /root/long-logs.py 100000 700

results in

$ docker logs xxxx | wc -c
1304

whose log size exceeds 50 x 2 = 100 bytes significantly.
1304 seems to be a full line 700 bytes + the last line 600 bytes (10000 % 700 = 600) with two "\r\n" characters appended by the API layer.

[achimnol]

Also, the log size applies to the console output, so depending on the logging driver, the actual filesystem size may vary. The differences may get quite large if the log size is big and when there are many short lines in the log. (e.g., an infinite loop of printing a single character line-by-line.)
This behavior should be documented or the implementation should be improved.

[cpuguy83]

I do not think there is any reason to try and make this so exact.
There is a 1MB line buffer.
We always write a full line (plus metadata like timestamp and which i/o stream it came from) to the log.

Having a log file stay smaller than 1MB and also write lines as large as 1MB does not seem like a real use-case.
Likewise having a container write lines that are larger than the max log file size does not seem like a real use case.

I'm going to close this as issues seem to be related to incorrect configuration. We have tests for this in tree as well.
If you are having this issue on Docker 19.03.6 please let us know the details.
And remember, containers must be re-created if you are relying on changes made daemon.json.
Thanks! 😇

[Dpz94]

This is not working even in the practical use case. Where I have given 50mb as max size and it still grows to GB.

[thaJeztah]

@Dpz94 have you verified the correct options are set on the container that has this issue (i.e., not just the daemon configuration)?

[Dpz94]

Am deploying my container using ansible, I have updated the daemon.json with above configuration and restarted the service and then created the container, any other additional steps required>??

My default log driver is json-file

[thaJeztah]

@Dpz94 I don't know what the Ansible deploy does w.r.t. configuration, so to find out what options the container was created with you need to do a docker inspect of the container and check its configuration.

[Dpz94]

Is there any particular option should we use while creating the container??, I edited the daemon.json file with the below config restarted docker service and deployed my container but still i could see my single log json file which is gone till 1 GB.

docker inspect on log config:

"LogConfig": {
"Type": "json-file",
"Config": {}

dameon.json:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "20"
}

[thaJeztah]

the LogConfig of that container shows that no options are set for the json-file logging driver, which means it does not have limits set.

When creating the container, either;

• set _no logging options_ (including _no_ logging-driver); in that case, the daemon defaults will be applied (which should be reflected in the container config after it's created (docker inspect <container-id>)
• or provide all custom options for the container (logging-driver + log-opts)

If you provide only the logging-driver, docker assumes you want to use that driver, and override the defaults that are set in the daemon configuration. If you don't provide log-opts for the container, it means: "reset the default limits that are set in the daemon configuration" (in other words: don't set any limits for this container).

[Dpz94]

@thaJeztah Thankyou so much, Am able to achieve it through custom options for the container but not able to make it use the daemon.json configuration I tried to create the container using logging option none but it is not taking the config from daemon.json rather it is showing as log driver none.If i create container without any log options it only taking the default driver json but not the config mentioned in daemon.json If that is not correct method of doing the set no logging options please assist. TIA

[MondaySam]

I'm sorry but we have the same issues with logging options not finding their way into the container.
we have set options in our docker-compose.yml but an inspect of the container shows empty log-opts.

We use

logging:
      driver: 'json-file'
      options:
        max-size: '1m'
        max-file: '5'

in our yml-file.

[thaJeztah]

@MondaySam Hm.. not able to reproduce that; I tried on:

docker-compose --version
docker-compose version 1.25.5, build 8a1c60f6

docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:21:11 2020
 OS/Arch:           darwin/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:16 2020
  OS/Arch:          linux/amd64
  Experimental:     true
...

With this docker-compose.yaml:

version: "3.7"
services:
  web:
    image: nginx:alpine
    logging:
      driver: json-file
      options:
        max-size: "1m"
        max-file: "5"

Deploy the compose-file with docker-compose:

docker-compose up -d
Creating network "repro-1148_default" with the default driver
Creating repro-1148_web_1 ... done

Inspect:

docker container inspect --format '{{ json .HostConfig.LogConfig }}' repro-1148_web_1 | jq .

{
  "Type": "json-file",
  "Config": {
    "max-file": "5",
    "max-size": "1m"
  }
}

Deploy the compose-file as swarm stack;

docker stack deploy -c docker-compose.yaml repro-1148

Creating network repro-1148_default
Creating service repro-1148_web

docker container inspect  --format '{{ json .HostConfig.LogConfig }}' repro-1148_web.1.jyyc9wo6iklxdk5sdrpsu9ogj | jq .

{
  "Type": "json-file",
  "Config": {
    "max-file": "5",
    "max-size": "1m"
  }
}

[MondaySam]

Nevermind. I'm stupid and wasn't fast enough to delete/revise my comment here.
Sorry to have bothered you.
Hope you are healthy and doing fine!

[HendrixSimon]

Hi,
I'm on Docker version 19.03.6 and am still experiencing the issue that docker would just log without constraints. Options are

logging:
    driver: json-file
    options:
        max-size: "50m"
        max-file: "3"

inspect shows:

"LogConfig": {
    "Type": "json-file",
    "Config": {
        "max-file": "3",
        "max-size": "50m"
    }
}

However, the log file still exceeds these limits and goes into gigabytes.
Anything I am missing?

[MondaySam]

The solution to my problem was rather simple.
Docker restart is simply not enough.
If the container was already running after you added the log rotation, you must use docker up -d again.

[HendrixSimon]

The container was completely recreated, but the issue still persists.
Also I updated to 19.03.11. Same issue.

[yifeikong]

I can confirm that you have to recreate the container to make existing containers work. This should be added to the docs.

docker version: Docker version 19.03.12, build 48a66213fe

[jvleminc]

Can confirm: Difference after container restarts, when on one container I had done a docker rm -f beforehand and on the other one not:

$ docker inspect prometheus_server| grep -B 0 -A 6 LogConfig
            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "3",
                    "max-size": "50m"
                }
            },
docker inspect nova_compute| grep -B 0 -A 6 LogConfig
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "host",
            "PortBindings": null,
            "RestartPolicy": {