Compose: [1.7] CTRL+C fails to stop containers when running compose in the foreground

[nickjj]

I can't reproduce it but I would say about 10% of the time when I hit CTRL+C, the containers that were started with docker-compose up fail to stop. I'll get an ABORT message in the terminal but they are still up. When it works, it says "gracefully stopping..." instead of ABORT.

At this point I need to docker-compose stop them to "really" stop them.

It was much worse in earlier versions of Compose but it's still happening often enough that I feel like it needs some attention. If you need more info let me know.

Environment

System

# Distro (it's actually xubuntu, not Ubuntu)
nick@oriath:~ ⚡ lsb_release --all
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:    14.04
Codename:   trusty

# Kernel
nick@oriath:~ ⚡ uname -a
Linux oriath 3.13.0-71-generic #114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

# Terminal info
nick@oriath:~ ⚡ xfce4-terminal --version
xfce4-terminal 0.6.3 (Xfce 4.10)

Docker

nick@oriath:~ ⚡ docker info
Containers: 4
 Running: 0
 Paused: 0
 Stopped: 4
Images: 187
Server Version: 1.11.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 300
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host null
Kernel Version: 3.13.0-71-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 5.877 GiB
Name: oriath
ID: 2P6Z:G7DQ:46LR:WO63:BDMP:ZAPH:XADM:AJNB:2WON:FTQD:RPIY:DYKH
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Username: nickjj
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Docker Compose

nick@oriath:~ ⚡ docker-compose --version
docker-compose version 1.7.0, build 0d7bf73

Potentially relevant information

When I encounter the issue I'm always running multiple containers, such as postgres, redis, and multiple web services. I never run 1 container so I can't say for sure if it would happen with only 1.

It happens with and without using links and both the legacy and v2 docker-compose.yml files.

[dnephin]

See #2904 for some background on the issue

[Vanuan]

Is it related to the docker's PID 1 problem?

[dnephin]

No, it's an issue with pyinstaller and signal handling.

[youcandanch]

Hm. I can make this happen reliably 100% of the time on a few of our repositories. If I wanted to help dig in with a fix, any pointers on where to start looking?

[toby-griffiths]

Any update on this issue, as I'm having the same problem with docker-compose version 1.9.0, build 2585387?

[tterrace]

Same issue on OSX docker-compose version 1.10.0, build 4bd6f1a.

[nickjj]

No, it's an issue with pyinstaller and signal handling.

I think so too.

I just had the same "ABORT" error happen to me while closing out the result of running docker-compose logs -f, so it's almost certainly related to the CTRL+C signal processing, and not so much the up command.

[dabroz]

Having the same problem on OSX 10.12.3:

docker-compose version 1.11.2, build dfed245

[vmalloc]

Also getting this on 1.12.0-rc2, build 08dc2a4

[tristanoneil]

Seeing this too. On OSX w/ Docker For Mac 17.05.0-ce-rc1-mac8 (16582). Details from docker-compose version.

docker-compose version 1.12.0, build b31ff33
docker-py version: 2.2.1
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2j  26 Sep 2016

[amirbehzad]

Same here under Darwin 16.6.0 and docker-compose version 1.11.2, build dfed245
^CERROR: Aborting.

[amirbehzad]

Here's a workaround:
https://github.com/docker/compose/issues/3317#issuecomment-209519731

[AlexanderOMara]

I'm also getting this issue randomly, about 30% of the time, using latest Docker for Mac on latest macOS.

The tty: true thing doesn't seem to have an effect after all. This is very confusing.

[cjerdonek]

In Dec. 2015, I offered a number of suggestions to make compose's signal handling code easier to check and reason about. This was in issue #2543 starting here.

A number of my suggestions were dismissed and don't look like they've been tried or given further consideration. Given that signal-handling continues to be an issue for this project (the issues today look more or less the same as before), I would suggest taking another look at what I suggested.

There are a few things I suggested back then that I think are worth considering in particular:

1. register your signal handlers at the very beginning of the program instead of "on the fly" in the middle of the program execution.
2. make your signal handlers return without raising an exception. This is because asynchronous exceptions have inherent problems in Python.
3. I also provided a pattern for handling a signal being sent more than once.

[gedimin45]

Hello,

Still happening in 1.17. Any plans to fix?

[camilo0365]

Still happening in version 1.20.0-rc2 . 😢

[ilyakamens]

Happening for me too.

$ docker-compose version
docker-compose version 1.18.0, build 8dd22a9
docker-py version: 2.6.1
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2j  26 Sep 2016

$ docker --version
Docker version 17.12.0-ce, build c97c6d6

OS X version 10.13.3 (17D102)

[bakku]

Here too on Docker for Mac:

docker-compose version 1.20.1, build 5d8c71b
docker-py version: 3.1.4
CPython version: 3.6.4
OpenSSL version: OpenSSL 1.0.2n  7 Dec 2017

[damouse]

Same on OSX 10.13.1
Docker Version 18.03.1-ce-mac65 (24312)
docker-compose version 1.21.1, build 5a3f1a3
docker-py version: 3.3.0
CPython version: 3.6.4

tty: true did not help.

[nhanft]

It's still happening :I
```Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:13:02 2018
OS/Arch: darwin/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:22:38 2018
OS/Arch: linux/amd64
Experimental: true

docker-compose version 1.21.1, build 5a3f1a3
docker-py version: 3.3.0
CPython version: 3.6.4
OpenSSL version: OpenSSL 1.0.2o 27 Mar 2018
```

[dylancopeland]

This seems to still be happening for me, too. Here's my setup:

docker-compose version 1.22.0, build f46880f
Docker version 18.06.0-ce, build 0ffa825
macOS 10.13.6 (17G2208)

[Vanuan]

Could you all please confirm what exactly is still happening when you hit CTRL+C?
I haven't seen "ABORT" message for a year.

tty: true doesn't have anything to do with the subject.

[dylancopeland]

After running docker-compose up and entering CTRL+C, here's what is output:

^CERROR: Aborting.

Running docker ps afterwards confirms that the containers are still running and were not properly stopped.

[Vanuan]

Ok, so there's still "an issue with pyinstaller and signal handling". Here's a statement from @dnephin :

pyinstaller raises thread.error for some unknown reasons (probably a bug in pyinstaller)

So this is an upstream issue, not in issue with docker-compose per se:
https://github.com/pyinstaller/pyinstaller/issues/208

[Vanuan]

Ok, so as a workaround we can install docker-compose from pypi:
https://pypi.org/project/docker-compose/#history

pip install docker-compose

[MythicManiac]

I'm working with Docker on two machines, both Windows and running version 18.06.1-ce-win73. On one machine Ctrl-C never kills the containers, whereas on the second machine it always works as intended.

If anyone has suggestions on what differences should I look for between the machines, I'm all ears.

[CSammy]

The issue may be fixed with https://github.com/pyinstaller/pyinstaller/pull/3515 which was merged in August, latest release of pyinstaller is v3.4 in September.

Without regressions, changing the file https://github.com/docker/compose/blob/master/requirements-build.txt should fix this issue if I am right. I'd prefer someone familiar with the project do a PR rather than myself though.

Edit: Note that pyinstaller needs to be called with --bootloader-ignore-signals as per the linked PR.

[cheungpat]

I gave the --bootloader-ignore-signals a try and it seems to have fixed the problem (at least for me). This is the the changes I made: https://github.com/cheungpat/compose/commit/a986cc9166d4d8992d38eca84d76bbfd06c85f70

Edit: I can make a PR but I want to see if this works for others as well.

[MythicManiac]

What fixed this for me was updating git bash on Windows, which is what I was running the docker commands on. My original issue seems unrelevant to the topic here, but maybe some folks with the same issue will end up here from google so posting anyway.

[bfirsh]

This is still the one issue I am running into daily using Compose.

https://github.com/docker/compose/pull/6518 has bumped pyinstaller to 3.4. I wonder if this fixes this, or do we have to add --bootloader-ignore-signals?

/cc @ulyssessouza @chris-crone

[ulyssessouza]

@bfirsh The change introduced by https://github.com/pyinstaller/pyinstaller/pull/3515 is activated only if --bootloader-ignore-signals is present, so the bump in #6518 didn't address to this issue.

The flag solution LGTM since the change looks functionally inoffensive and passed the tests on my machine.

@cheungpat would you like to propose a PR based on https://github.com/cheungpat/compose/commit/a986cc9166d4d8992d38eca84d76bbfd06c85f70 ?

[torarnv]

Upgraded to 1.24.0 with the curl instructions in https://github.com/docker/compose/releases/tag/1.24.0

I'm still seeing this:

home-assistant    | 2019-06-23 20:58:55 INFO (SyncWorker_1) [homeassistant.loader] Loaded mqtt from homeassistant.components.mqtt
^CERROR: Aborting.

❯ docker-compose -v
docker-compose version 1.24.0, build 0aa59064

[torarnv]

Adding tty: true does not seem to help

[chris-crone]

@torarnv I'm not sure that the fix was merged into 1.24. Can you try with 1.25.0-rc1?

[gromilQaaaa]

1.25 has the same problem on ubuntu for me