Compose: "docker-compose logs -f" stops outputting logs of service after some time

[Greek64]

Currently I'm running a Docker Compose project with 20 services.
One of these services is "gitlab/gitlab-ce:latest", which is quite chatty in it's logging output.
[about 80 lines/min]

I always start my services with "docker-compose up -d" followed by a "docker-compose logs -f"
The Problem is, that after a (until now) unspecified time the logging of the "gitlab" container/service is not outputted with "docker-compose logs -f". But investigation of the actual logfile of the container/service shows that new log output is generated, but somehow ignored by "logs -f".
A termination of the command (via CTRL+C) and a new execution of the command "docker-compose logs -f" shows and follows again the output of the "gitlab" container/service for an unspecified time.

I realize that this is a specific scenario that I'm presenting. I will therefore try to find a minimal reproducible example of this behavior.

Additional Info:

docker-compose version 1.21.2, build a133471
docker-py version: 3.3.0
CPython version: 3.6.5
OpenSSL version: OpenSSL 1.0.1t  3 May 2016

Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:10:06 2018
 OS/Arch:       linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:08:35 2018
  OS/Arch:      linux/amd64
  Experimental: false



md5-0cbd690ae469a185e8ef6ef6834ed0c2



Debian GNU/Linux 9 (stretch)
4.9.0-6-amd64 #1 SMP Debian 4.9.88-1 (2018-04-29) x86_64 GNU/Linux

[Greek64]

Possible related issue: #4263

[hwh33]

I've got the same issue and have for some time.

It appeared to be related to #3633 which was closed as related to Moby issue 35933. Following advice in issue 3633 and setting DOCKER_COMPOSE_TIMEOUT to a large value worked as a short-term fix.

The Moby issue referenced was resolved for Docker 18.03.1, but I'm still experiencing the same issue as @Greek64. Additionally, setting DOCKER_COMPOSE_TIMEOUT to a large value no longer seems to work as a short term fix.

My info:

docker-compose version 1.21.1, build 5a3f1a3
docker-py version: 3.3.0
CPython version: 3.6.4
OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:13:02 2018
 OS/Arch:      darwin/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:22:38 2018
  OS/Arch:      linux/amd64
  Experimental: false

System Version: macOS 10.13.5 (17F77)
Kernel Version: Darwin 17.6.0

[zwhitchcox]

so, sorry, but where do you input DOCKER_COMPOSE_TIMEOUT? Do you export the variable to the bash environment, I'm guessing?

[zwhitchcox]

Oh, sorry, I misread that, I see that doesn't work anymore.

[zwhitchcox]

I'm having this same issue on

docker-compose 1.23.2, build 1110ad0
docker 18.09.0, build 4d60db4

client:
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

Server: All servers don't work, including a node server, golang server, adminer, postgres, none of them output anything.

I had to downgrade from 18.10 for an issue with docker...maybe I need to downgrade back to 16.04?

[zwhitchcox]

So, presumably someone has these logs working...if they do, can they tell me what version they're doing so I can use that version?

[zwhitchcox]

I was able to get it working by going to 1.23.1

[Greek64]

Just to clarify, I still have this issue and I too am currently on
docker-compose version 1.23.2, build 1110ad01
docker version 18.09.0

Server: All servers don't work, including a node server, golang server, adminer, postgres, none of them output anything.

So, are no logs generated at all, is docker log not showing up anything or is docker-compose logs -f seizing to display any output after a while? The latter is the cause of this issue. if you have a different phenomenon I would suggest opening a separate issue.

Further information regarding this issue:

It has now happened to many different services, all of which generate more output than their counterparts which continue to display their output via the logs -f command.
It is especially prone to happen when one does a docker-compose logs -f after the containers have lived for a while (which means that all their past logs all displayed at once).
This suggests that this issue is probably due to a race condition, which I haven't figured out yet.

[zwhitchcox]

I don't know if my issue is weird, but docker-compose logs worked fine for me. And when all the logs stop, they all stop at the same time.

But I was able to fix the issue by rolling back to 1.23.1, but yeah, 1.23.2 appears to have broken that functionality for me.

I have no idea what could have caused it, and it looks like you have your work cut out for you though, good luck.

[pasisavolainen]

This was killing me and docker-compose logs has some uninteresting services logged and also failed hard handling '^M' (CRLF line-endings) on Docker4Windows, so now there is this: https://github.com/pasisavolainen/tools#dockertailpy Follows only containers you want to, renames them and doesn't stop when container is recreated. But it's slowish (docker logs for each container), so pick your poison.

[Greek64]

docker-compose logs has some uninteresting services logged [...]
Follows only containers you want to [...]

This you could handle also with docker-compose by doing following
docker-compose logs -f service1 service2 service3 ...

[pasisavolainen]

This you could handle also with docker-compose by doing following
docker-compose logs -f service1 service2 service3 ...

with names like "dir_somequalityprocessingservice_abscde12"; naah :)

[renevall]

I'm experiencing this right now

docker-compose version 1.23.2, build 1110ad01
docker-py version: 3.6.0
CPython version: 3.6.7
OpenSSL version: OpenSSL 1.1.0f 25 May 2017

[JamesBalazs]

Also experiencing this in 1.23.2, but can't go lower than 1.23.2 and must be lower than 1.24 for the AWS EB CLI, so I'm stuck on this version with no minor version patch

[feltech]

Experienced on 1.24 and also experienced after downgrading to 1.23.1, as suggested above. Specifically I'm trying to use docker-compose up to show logs from a stack on an Ubuntu 16.04 dev workstation with versions

docker==3.7.2
docker-compose==1.23.1
docker-ce:
  Installed: 5:18.09.3~3-0~ubuntu-xenial

I traced the problem to that fact that if no logs come in within COMPOSE_HTTP_TIMEOUT seconds (defaults to 60) then all logs stop, forever. This seems to be alluded to in https://github.com/docker/compose/issues/3633#issuecomment-365101768 ("Unresponsive container operations ... being addressed at the engine level ... https://github.com/moby/moby/issues/35933", but the linked issue is frankly TLDR).

[divmgl]

This is still a problem. Currently happening very frequently when using nodemon. Seems to happen when the log stays idle for some time.

[Greek64]

Just want to inform that I found the cause for my problem.
I had configured the docker logging options to max-size: 5m and max-file: 1, and every time the logfile reached the max size and was rotated, the trailing of docker-compose logs -f would just stop/hang.
Note that I do not have a timeout issue.

Turns out this is not a docker-compose specific issue, but a docker issue in general, as it also happens with docker logs -f.
Here is a related issue in the docker repo

My fix that works until now is to just increase the max-file setting to at least 2.

I will leave this issue still open until I can confirm this issue/fix for sure, and link an issue to the docker repository.

[Greek64]

Issue in docker repo

[ijc]

Thanks @Greek64 for reporting back. Since https://github.com/moby/moby/issues/39235 seems to be in hand, and is any case a better place to track this issue I'm going to close this issue.

To anyone else who is seeing the same symptom, please check if you are affected by the engine issue @Greek64 has identified. If your issue wasn't caused by that same underlying issue I would suggest opening fresh issues so we can track different problems which happen to cause similar symptoms separately rather than in one big issue which will soon get unwieldy.