Compose: Feature: Ability to clear log history

[djessup]

A feature I've thought would be useful since originally using Fig, and now Compose would be the ability to clear the log history for Composed-managed containers. Long-running or "chatty" containers can end up with a lot of log noise that may not be wanted.

I'd expect a command like the following would solve the problem:
$ docker-compose logs --clear [service]

[dnephin]

I don't think that is a feature supported by the docker daemon. Looking at the api docs, the only option would be to truncate the returned logs to a limited number of lines per container:

https://docs.docker.com/reference/api/docker_remote_api_v1.17/#get-container-logs

[metal3d]

:+1: for this issue. I actually use compose to develop a website in Golang, Mongodb and nginx... 5 days after begining I have a long logs that becomes to be worrying. Each time I restart containers append many lines to the log.
@dnephin I don't understand if you give a solution (that I don't understand :)) or if you propose to check if it's possible with api. Sorry for my bad english.

[thaJeztah]

Docker 1.6 will add support for logging drivers, see https://github.com/docker/docker/pull/10568 (currently; JSON, syslog and "none") work is in progress for basic log-rotation; https://github.com/docker/docker/pull/11485

[metal3d]

Good to hear that, thanks a lot :)

[djessup]

Thanks for the background and update on Docker 1.6 - looking forward to it!

[sgarbesi]

docker logs -c (clear) <container> would be great.

+1

[baptistedonaux]

+1 Really important

[egyptianbman]

Holy cow, I just sat through several minutes of logs to get to the end. I'd really appreciate this as well for the sake of not having to rebuild containers constantly.

[jghibiki]

+1

[vronin]

+1

[junajan]

+1

[knutole]

:+1:

[Rodeoclash]

+1

[nafg]

+2

[FnTm]

+1

[dmage]

Just for the record, with Docker 1.8 and docker-compose 1.4 there is already exists a method to limit log size using https://docs.docker.com/compose/yml/#log-driver and log-opt max-size:

  log_driver: "json-file"
  log_opt:
    max-size: "100k"
    max-file: "20"

[knutole]

@dmage Thanks, exactly what I need.

[OlivierCuyp]

+1000

[sheyman]

+1 would LOVE this

[joegoggins]

+1

[Rodeoclash]

@dmage solution works great for us. Those others who are +1ing this, what is wrong with the solution that he's given?

[rosskevin]

@Rodeoclash - I think one use case for this is reuse of a given set of containers for a new run i.e. CI test run. The old logs are irrelevant to the new run, so a clear prior to the next command would clear up confusion.

[SpeCT]

+1

[ballinette]

+1, same need as @rosskevin

[dnephin]

Ok, to recap:

• you can do this from a compose file (see https://github.com/docker/compose/issues/1083#issuecomment-141936600)
• #265 covers being able to limit the output from the logs command
• #1756 covers the re-used container case

Going to close this since it's already support or tracked in other issues.

[chris-martin]

I don't quite understand why this was closed. How do you clear the log history?

[egyptianbman]

I add:

  log_opt:
    max-size: 50k

To limit the length of the logs.

[sgarbesi]

They really need to add this, it's essential. Limiting logs is good and all but there should be a simple command to clear logs.

docker logs -c <container>

What gives?

[chris-martin]

I understand how to limit the log size, but how do you _clear_ the log?

[dnephin]

I don't think clearing a log is supported by the docker engine, which is what managed the logs.

Maybe with a custom log driver you could do that, but that would be external to compose.

[geekscrapy]

+1 for being able to flush the logs...

[kongchen]

+1

[ObjReponse]

+1

[kierr]

+1

[mjaverto]

+1

[helinwang]

+1 for clear log command.

[dgommeren]

+1 for clear log command

[max3-05]

+1

[Jarema]

+1

[mskutin]

+1

[kytwb]

:+1:

[emanuil]

+1

[mcuadros]

+1

[wavenator]

Take a look at that doc:
https://docs.docker.com/engine/admin/logging/overview/

[sgarbesi]

docker-logs-clean.sh

#!/bin/bash

rm $(docker inspect $1 | grep -G '"LogPath": "*"' | sed -e 's/.*"LogPath": "//g' | sed -e 's/",//g');

---

Invocation:

sudo ./docker-logs-clean.sh <container-name>;

[impanda-cookie]

@sgarbesi thanks!
+1

[ariden]

+1 for clear log command

[paolomainardi]

Closed in favor of ?

[dnephin]

To reiterate my comment from https://github.com/docker/compose/issues/1083#issuecomment-149357280:

Ok, to recap:

• you can do this from a compose file (see https://github.com/docker/compose/issues/1083#issuecomment-141936600)
• #265 covers being able to limit the output from the logs command
• #1756 covers the re-used container case

If what you're looking for is some command likedocker logs --clear, that is not supported by docker engine, so you would need to request that on docker/docker. However, I think the above options should already be sufficient for most cases. What most people want is to just show a subset of the logs, not actually remove them.

[djessup]

Thanks @dnephin, and everyone else for their contributions. I raised this issue almost a year ago now, and judging from the comments since then it seems pretty clear to me that log management is a pain point for a lot of compose users.

There have been a few workarounds mentioned, myself I mostly use max-size to keep the logs to a reasonable length. This helps a lot, and I'm grateful for these workarounds, but it's important to keep in mind this is a _workaround_, not a solution.

It's also clear to me now that part of the responsibility for fixing this lies within the Docker logging system and that it would need to provider a clear command for Compose to utilise - fair enough.

All that said however, there are a few features that have made it into Docker releases since this ticket was created - namely --since=<timestamp> and --tail=<num-lines> that could be supported by Compose to come closer to giving a real solution to this issue.

For example, supporting --since could make something like this possible:

$ docker-compose logs --since=now my_container

or

$ docker-compose logs --since=5m my_container

Support for --tail would also be useful, e.g.

$ docker-compose logs --tail=100 my_container

and of course, combinations thereof. It may also make sense to support these in docker-compose.yml as part of the logging block, but even without that just by supporting these two options I suspect you could satisfy the majority of people who've +1'd this ticket.

In summary, thanks again for all the feedback and workarounds, and for Docker and Compose themselves - they're both great products - I hope you'll consider the ideas put forward in this thread and continue to make these products even greater.

[dnephin]

See #2227 for a summary

[djessup]

:+1: thanks @dnephin, looking forward to it!

[samarti]

What I did was to manually delete the <container-id>-json.log files placed inside /var/lib/docker/containers/<container-id>/ (use sudo). Once I run docker-compose logs, the log was empty. It's not a solution but with a proper .bash file you can automatize the cleaning before each build.

Edit: Something like this does the trick (use under your own risk!!):

sudo find /var/lib/docker/containers/ -type f -name '*-json.log' -delete

[douineauromain]

+1

[godely]

+1

[zakariaboualaid]

+1

[cmtec]

+1

[vhosakot]

+2

[joshbedo]

+2

[eexit]

+1

[vivex]

+20

[iagopiimenta]

+1

[JohnWelle]

+1

[qub]

+1

[pbn04001]

+1

[kassanmoor]

@sgarbesi After run that clean command, the log feature will work normally?

[qrpike]

+1

[lvitals]

docker-logs-clean.sh

#!/bin/bash

for container_id in $(docker ps -a --filter="name=$name" -q);

    do file=$(docker inspect $container_id | grep -G '"LogPath": "*"' | sed -e 's/.*"LogPath": "//g' | sed -e 's/",//g');

    if [ -f $file ]
      then
          rm $file;
    fi

done

Invocation:

chmod +x docker-logs-clean.sh

sudo ./docker-logs-clean.sh

[vinogradov]

+1 for command line option for clearing logs manually

[sgarbesi]

@kassanmoor Does for me.

[Nicolas-Leucci]

+1

Thanks for other ways to clean container logs

[mirkods]

👍

[HerbertChu]

+1 👍

[krzysztof-kabala]

+1 !

[dottgonzo]

+1

[paustin01]

+1

[alexlenail]

+1

[wazoo]

This would be a great feature to have, I have a few containers that are Java based and after even a day or two when troubleshooting it can take 30 seconds or more to do a docker logs -f <container> because Java loves those multi-line logs.

Another thing that might be easier to implement is another flag on logs to start a tail without echoing all of the existing logs docker logs -f -n <container> for example (meaning follow only new logs). This would only echo log messages received after running the command.

The opposite would also work (and would be closer to the way that gnu tail works) is doing a -f echos the last 5-10 lines and then new lines by default and adding a flag to echo everything as it does now (maybe docker logs -f -a <container> or something).

On that note I would love to have the feature from gnu tail that allows specification of how many lines to tail like docker logs -100 <container> gives me the last100 lines.

[kopax]

+1

[holygits]

+1

[michaelfreund]

+1

[ismay]

+1

[bjspatel]

+1

[fatcloud]

+1

[pvoisin]

+1

[Valten1992]

+1

[twooster]

+1

[shuklara]

+1

[ahfeel]

Is there a trick for Docker for mac ? Thanks and +1 for a command line option for this :)

[alirezaDavid]

+1

[romanesko]

+1

[Manoj-nathwani]

+1

[DavidPesticcio]

UPDATE: 2016/10/08 - Removed requirement for "jq" as "docker logs" supports Go templates! (https://docs.docker.com/engine/admin/formatting/)

Hi all,

Because deleting files that could be open is generally a BAD idea - especially if you are trying to save diskspace! - I have expanded upon the initial efforts of @sgarbesi, @lvitals & @wazoo (thanks for the ideas guys) to produce the slightly more functional script below.

Copy the code below into a file
vi ./docker-container-log-trim.sh
Make the file executable
chmod +x ./docker-container-log-trim.sh
Then execute with
sudo ./docker-container-log-trim.sh

Uncomment the line with the > when you trust the script... that's where the magic happens. :-)

Feedback welcome.
Thanks.

PS. Tested, but not battle-hardened. Use at own risk.

#!/bin/bash

# NOTES:
#  Does NOT delete logfile (BAD IDEA) - simply trims file with redirect.
#  Handles single/all-running/all-existing containers - see end of script for usage.
#  Enjoy :-)


_get_container_logfile() {

  case $1 in

    running) _trim_container_logfile "$(docker ps -q)" $2
             ;;

        all) _trim_container_logfile "$(docker ps -aq)" $2
             ;;

          *) _trim_container_logfile "$(docker ps -a | awk -v ID=$1 '$1 ~ ID || $NF ~ ID {print $1}')" $2
             ;;

  esac

}


_trim_container_logfile() {

  TEMP=$(mktemp)

  case $2 in
    *[!0-9]*) echo "[lines] must be a number - \"$2\" is not a number."
              exit 1
              ;;
        ''|*) MAX=${2:-1000}
              ;;
  esac


  if [ -z $1 ]
  then
    echo "Container name/id unknown!"
    exit 1
  else
    for container in $1
    do
      logfile="$logfile $(docker inspect --format '{{ .LogPath }}' $container)"
      echo "Keeping $MAX lines: $logfile"

      tail -n ${MAX} $logfile > $TEMP
      # Uncomment the next line when you trust the script!
      # cat $TEMP > $logfile
    done
  fi

  rm $TEMP
}


if [ -a "$(which docker)" ]
then
  case $1 in
    --trim) if [ -z $2 ]
            then
              echo "Container name/id missing!"
              exit 1
            else
              _get_container_logfile $2 $3
            fi
            ;;

    --trim-running) _get_container_logfile running $2
                    ;;

    --trim-all) _get_container_logfile all $2
                ;;

    *) echo "Usage:"
       echo "  --trim {container} [lines]   Keep [lines] of logfile for a single container"
       echo "  --trim-running     [lines]   Keep [lines] of logfile for all running containers"
       echo "  --trim-all         [lines]   Keep [lines] of logfile for all containers"
       echo "Default: lines=1000"
       exit 1
       ;;
  esac
else
  echo "Requires \"docker\""
  exit 1
fi

[babasaikiran]

+1

[xiemeilong]

+1

[etc0de]

So since it was expressed multiple times that there is a wish for an explicit clear command and since there was never one added, any chance this will be reopened?

It depends on a docker engine change of course, but it's still something that will eventually need to be addressed in docker-compose as well - and it's definitely not fixed as far as an explicit command is concerned.

[gingerlime]

@DavidPesticcio I'm getting this error when running your script: line 53: $logfile: ambiguous redirect (after removing the comment)

[DavidPesticcio]

Hi @gingerlime, Seems like $TEMP is not being populated... maybe you don't have "mktemp" installed, or it's not in your path perhaps? :-/

"Works fine for me" - yeah I know, that's not much help to you, but it's true... :-)

I have updated the script, so you don't need "jq" any more - perhaps I should add a bail-out if mktemp is missing too... I thought it was a standard tool - but maybe you're not running the script within a "standard" install - like from within a minimal container maybe? :-)

Hope that helps!

[gingerlime]

I do have mktemp, and use it frequently...Didn't really spend much time debugging it. Ended up with the simpler script above that just nukes those logs. In our dev environment they are not important.

[fruch]

on my setup (i.e. running as no root user) this bash script isn't helping casue i'm getting premission denied trying to open the log file.

it's a bit weird, that as a user i can start docker, but can tocuh it's log files...
more of a reason to have something expend to the docker/docker-compose commands

[gamma]

I had to modify the script of @DavidPesticcio because of the errors mentioned before ... here it goes:

#!/bin/bash

# NOTES:
#  Does NOT delete logfile (BAD IDEA) - simply trims file with redirect.
#  Handles single/all-running/all-existing containers - see end of script for usage.
#  Enjoy :-)


_get_container_logfile() {

  case $1 in

    running) _trim_container_logfile "$(docker ps -q)" $2
             ;;

        all) _trim_container_logfile "$(docker ps -aq)" $2
             ;;

          *) _trim_container_logfile "$(docker ps -a | awk -v ID=$1 '$1 ~ ID || $NF ~ ID {print $1}')" $2
             ;;

  esac

}


_trim_container_logfile() {

  TEMP=$(mktemp)

  case $2 in
    *[!0-9]*) echo "[lines] must be a number - \"$2\" is not a number."
              exit 1
              ;;
        ''|*) MAX=${2:-1000}
              ;;
  esac


  if [ -z "$1" ]
  then
    echo "Container name/id unknown!"
    exit 1
  else
    for container in $1
    do
      logfile="$(docker inspect --format '{{.LogPath}}' $container)"
      if [ ! -f "$logfile" ]; then continue; fi
      echo "Keeping $MAX lines: $logfile"

      tail -n ${MAX} "$logfile" > "$TEMP"
      # Uncomment the next line when you trust the script!
      # cat "$TEMP" > "$logfile"
    done
  fi

  rm "$TEMP"
}


if [ -a "$(which docker)" ]
then
  case $1 in
    --trim) if [ -z $2 ]
            then
              echo "Container name/id missing!"
              exit 1
            else
              _get_container_logfile $2 $3
            fi
            ;;

    --trim-running) _get_container_logfile running $2
                    ;;

    --trim-all) _get_container_logfile all $2
                ;;

    *) echo "Usage:"
       echo "  --trim {container} [lines]   Keep [lines] of logfile for a single container"
       echo "  --trim-running     [lines]   Keep [lines] of logfile for all running containers"
       echo "  --trim-all         [lines]   Keep [lines] of logfile for all containers"
       echo "Default: lines=1000"
       exit 1
       ;;
  esac
else
  echo "Requires \"docker\""
  exit 1
fi

[etc0de]

@dnephin any chance this could be reopened, since there is an obvious demand to have an explicit one-time clear command for logs?

[oogali]

Truncate the logs for a given container (must be root):

cp /dev/null $(docker inspect -f '{{.LogPath}}' container_name)

You want truncation, not deletion. (Deleting the file referenced by an open file handle doesn't reclaim space until the process, in this case, the Docker daemon, completely exits)

[etc0de]

@oogali fair enough. Still would be nice to have a proper command for it to do it on demand.

[lpoulter]

+1

[ghost]

would be nice to have a proper command for it

[j2l]

+1
would be nice to have a proper command for it to do it on demand.

[nickwinn]

This is a no brainer, setting a start date as a filter is more work then just clearing it to look at current output. Please add this feature.

[erikverheij]

+1

[MichaelErmer]

+1

[heytrav]

+1

[piotr-s-brainhub]

+1

[eduardoleal]

+1

[Joshfindit]

Tangentially related suggestion:

When doing docker-compose logs -f, automatically default to --tail=30 (or any other number that's reasonable)

[piotr-s-brainhub]

docker-compose logs -f is not enough because with large number of logs displaying it takes a very long time

[phlegx]

+1

[luchesar]

+1

[edwardmp]

+1

[rabahs]

+1

[htdttb]

+1

[atedja]

+1

[ndoulgeridis]

+1

[ahmetkucuk]

+1

[sychevsky]

+1

[OzzyCzech]

+1

[JeanLescut]

Or there is this workaround also

logpath=`docker inspect --format='{{.LogPath}}' reveelium_metricsextraction_1` && mv $logpath $logpath".bckup"

[ReFruity]

Please, stop commenting +1. It makes this thread really hard to read and extract valuable information. There is a thumbs up button on OP post for this.

[avindra]

Nothing here works for me with Docker for mac. I've hacked together something that works based on reading this thread and the docker forums however.

The problem with D4M, is that on mac, you need to actually run the commands on the xhyve vm. So here's what I came up with. Add these two functions to your .bash_profile.

Important: dont forget to start a new shell or reload your profile before continuing.

---

Now, docker-logs-clean looks like this:

#!/bin/bash -e

if [[ -z $1 ]]; then
    echo "No container specified"
    exit 1
fi

logFile=$(docker inspect -f '{{.LogPath}}' $1 2> /dev/null)

echo -n "Cleaning ${logFile}... "
d4mexec << EOF
> $logFile
EOF
echo "done"

Note that I am not rming the log file, but just doing >, which will fully truncate the file.

[tgjamin]

FWIW in docker-compose version 2 the feature to limit log file sizes:

version: '2'
services:
  my-service:
    image: nginx:alpine
    restart: always
    logging:
      # limit logs retained on host to 25MB
      driver: "json-file"
      options:
        max-size: "500k"
        max-file: "50"

It is not documented on the docker website very well, and this may be useful to others.

[heetap]

+1

[sintetico82]

+1

[d13g0s0uz4]

+1

[ccamacho]

+1

[MPJHorner]

+1

[WhiteBearSpirit]

I made a simple script docker_clear_log.sh:
sudo truncate -s 0 $(docker inspect --format='{{.LogPath}}' $1)
Usage: ./docker_clear_log.sh [Name-or-ID]
Should work if you have sudo rights and log_driver: "json-file" setting for your docker (default).

[gsugambit]

+1

I also tried to temp solution and it appears to work

[educatedwarrior]

The original issue was never resolved and the issue was closed anyway. Interesting. @djessup , how do you like those apples?

[dhakehurst]

would still be nice to have a docker-compose logs --clean

[linvi]

Why has this issue been closed without even a comment when the issue in itself has not been solved?

[Ghoughpteighbteau]

@linvi

When I came to this thread, the closing comment had been folded into other comments. It's right here:

https://github.com/docker/compose/issues/1083#issuecomment-149357280

I think they're hoping to cover this particular use case through indirect means. Mention you're particular use case and how it's not covered, and it might help the ticket be re-opened. Worst case, someone might point out an easy way to get what you want :D

[MartinNiederl]

Any updates?

[FelixFurtmayr]

when there is a log, we should be able to clear it in a simple way.

would be really nice, if you could build the command into it. this seems not hard.
thousands of people would be very happy not having to scroll through everything.

[FelixFurtmayr]

If you use docker-compose, use lazydocker for viewing logs. Then there is then no need to delete logs. You get live logs and start with the latest logs. This helps very much with debugging.

https://github.com/jesseduffield/lazydocker
You can install it with one line and it is a good monitoring tool. It is a pitty that people are not so reasonable to understand that a "delete log" function is a good thing.

But the workaround with lazy docker does it for me. Thank you jesseduffield for giving us the possibility to be lazy with your monitoring tool :-)
Maybe consider donating to lazydocker if this makes debugging/monitoring for you also easier.
And for you docker developers / maintainers: why is the interface from docker not like that?
Docker is pretty awesome, but have a look at lazydocker; there is room for improvement.

[GF-Huang]

[phone]

[kagansari]

[phone]