Compose: Secsh channel 10 open FAILED: open failed: Connect failed

[Vmadmax]

Description of the issue

I don't know exactly if this is a docker-compose bug or docker bug.

Connection faild on docker-compose -H 'ssh://[email protected]' up
Without the host parameter, the docker-compose will start without problems.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.24.0-rc1, build 0f3d4dda
docker-py version: 3.7.0
CPython version: 3.6.6
OpenSSL version: OpenSSL 1.1.0h  27 Mar 2018

Output of docker version
Local:

Client: Docker Engine - Community
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:33:12 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:41:49 2019
  OS/Arch:          linux/amd64
  Experimental:     true

Remote:

Client:
 Version:           18.09.1
 API version:       1.38 (downgraded from 1.39)
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:31 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       e68fc7a
  Built:            Mon Oct  1 14:25:33 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker-compose config

services:
  app:
    build:
      context: /Users/xx/xx/project
      dockerfile: app.dockerfile
    environment:
      DB_HOST: database
      DB_PORT: '3306'
    volumes:
    - /Users/xx/xx/project:/var/www:rw
    working_dir: /var/www
  database:
    environment:
      MYSQL_DATABASE: xxx
      MYSQL_PASSWORD: xxx
      MYSQL_ROOT_PASSWORD: xxx
      MYSQL_USER: xxx
    image: mariadb:10.1.26
    ports:
    - 33061:3306/tcp
    volumes:
    - dbdata:/var/lib/mysql:rw
  web:
    build:
      context: /Users/xx/xx/project
      dockerfile: web.dockerfile
    ports:
    - 8080:80/tcp
    volumes:
    - /Users/xx/xx/project:/var/www:rw
    working_dir: /var/www
version: '3.0'
volumes:
  dbdata: {}

Steps to reproduce the issue

1. docker-compose -H 'ssh://[email protected]' up

Expected result

Creating huelle_app_1      ... done
Creating huelle_database_1 ... done
Creating huelle_web_1      ... done

Stacktrace / full error message

 docker-compose -H 'ssh://[email protected]' up
Connected (version 2.0, client OpenSSH_7.6p1)
Authentication (publickey) successful!
Creating huelle_app_1 ...
Creating huelle_database_1 ...
Creating huelle_web_1      ...
ERROR: Secsh channel 10 open FAILED: open failed: Connect failed

ERROR: for huelle_app_1  (2, 'Connect failed')
ERROR: Secsh channel 11 open FAILED: open failed: Connect failed

ERROR: for huelle_web_1  (2, 'Connect failed')
ERROR: Secsh channel 12 open FAILED: open failed: Connect failed

ERROR: for huelle_database_1  (2, 'Connect failed')

ERROR: for app  (2, 'Connect failed')

ERROR: for web  (2, 'Connect failed')

ERROR: for database  (2, 'Connect failed')
Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 127, in perform_command
  File "compose/cli/main.py", line 1085, in up
  File "compose/cli/main.py", line 1081, in up
  File "compose/project.py", line 555, in up
  File "compose/parallel.py", line 112, in parallel_execute
  File "compose/parallel.py", line 210, in producer
  File "compose/project.py", line 541, in do
  File "compose/service.py", line 533, in execute_convergence_plan
  File "compose/service.py", line 455, in _execute_convergence_create
  File "compose/parallel.py", line 112, in parallel_execute
  File "compose/parallel.py", line 210, in producer
  File "compose/service.py", line 453, in <lambda>
  File "compose/service.py", line 441, in create_and_start
  File "compose/service.py", line 337, in create_container
  File "compose/container.py", line 52, in create
  File "site-packages/docker/api/container.py", line 427, in create_container
  File "site-packages/docker/api/container.py", line 437, in create_container_from_config
  File "site-packages/docker/api/client.py", line 284, in _post_json
  File "site-packages/docker/utils/decorators.py", line 46, in inner
  File "site-packages/docker/api/client.py", line 221, in _post
  File "site-packages/requests/sessions.py", line 572, in post
  File "site-packages/requests/sessions.py", line 524, in request
  File "site-packages/requests/sessions.py", line 637, in send
  File "site-packages/requests/adapters.py", line 449, in send
  File "site-packages/urllib3/connectionpool.py", line 600, in urlopen
  File "site-packages/urllib3/connectionpool.py", line 354, in _make_request
  File "http/client.py", line 1239, in request
  File "http/client.py", line 1285, in _send_request
  File "http/client.py", line 1234, in endheaders
  File "http/client.py", line 1026, in _send_output
  File "http/client.py", line 964, in send
  File "site-packages/docker/transport/sshconn.py", line 29, in connect
  File "site-packages/paramiko/transport.py", line 806, in open_session
  File "site-packages/paramiko/transport.py", line 944, in open_channel
paramiko.ssh_exception.ChannelException: (2, 'Connect failed')
[4866] Failed to execute script docker-compose

Additional information

OS version local: macOS Mojave, 10.14.2 (18C54)
OS version remote: Ubuntu 18.04.1 LTS

[chris-crone]

Thanks for reporting this @Vmadmax!

[rumpl]

Hey @Vmadmax,

I was able to reproduce this error.

The issue is that compose opens too much sessions per connection, ssh limits them to 10 by default. A temporary workaround would be to change this value in your remote sshd configuration (the MaxSessions parameter) and set it to ~30. We are working on a fix for this.

Please note that you need Docker 18.09 if you want to connect to your daemon over SSH, your remote server has 18.06.1-ce.

[Vmadmax]

Hi @rumpl,
thanks for your feedback.

I updated to the last version and set the MaxSessions to 100 - it's working, but this is happening now:
docker-compose -H 'ssh://[email protected]' up

Building app
Step 1/16
....
Successfully built 7979ea4ad8de
Successfully tagged huelle_app:latest

Building web
Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 127, in perform_command
  File "compose/cli/main.py", line 1085, in up
  File "compose/cli/main.py", line 1081, in up
  File "compose/project.py", line 527, in up
  File "compose/service.py", line 344, in ensure_image_exists
  File "compose/service.py", line 1084, in build
  File "site-packages/docker/api/build.py", line 268, in build
  File "site-packages/docker/utils/decorators.py", line 46, in inner
  File "site-packages/docker/api/client.py", line 221, in _post
  File "site-packages/requests/sessions.py", line 572, in post
  File "site-packages/requests/sessions.py", line 524, in request
  File "site-packages/requests/sessions.py", line 637, in send
  File "site-packages/requests/adapters.py", line 449, in send
  File "site-packages/urllib3/connectionpool.py", line 600, in urlopen
  File "site-packages/urllib3/connectionpool.py", line 354, in _make_request
  File "http/client.py", line 1239, in request
  File "http/client.py", line 1285, in _send_request
  File "http/client.py", line 1234, in endheaders
  File "http/client.py", line 1026, in _send_output
  File "http/client.py", line 964, in send
  File "site-packages/docker/transport/sshconn.py", line 29, in connect
AttributeError: 'NoneType' object has no attribute 'open_session'
[15517] Failed to execute script docker-compose

[rumpl]

Could you paste the complete output of docker-compose --verbose -H 'ssh://[email protected]' up please?

[tobyfoo]

I can also reproduce the AttributeError: 'NoneType' object has no attribute 'open_session' error. Some observations below plus docker-compose --verbose logs attached.

• It happens only when/after an image is built using the "up" command. The image is still built successfully.
• The first docker-compose -H 'ssh://...' up always fails because an image needs to be built. Repeating the command works with no error as the image now already exists on the remote host.
• docker-compose -H 'ssh://...' up --build always produces the error (but the image is still built).

Attached log file of this call, docker-compose --verbose -H "ssh://ubuntu@remotehost" up --build &> docker-compose-up-build.log using docker-compose version 1.24.0-rc1 (build 0f3d4dda) locally and Docker version 18.09.1 on the remote host.

Thanks!
docker-compose-up-build.log

Edit:
Remote host sshd MaxSessions was set to 500.
As a workaround we are first calling docker-compose -H 'ssh://...' build and then docker-compose -H 'ssh://...' up. This does not cause the error.

[rumpl]

Thanks @tobyfoo!

I opened an issue on docker-py, the SSH transport is not behaving like the others.

Hopefully we will have a fix soon.

[arcenik]

HI.

I've made a PR for that on docker-py: https://github.com/docker/docker-py/pull/2262

But it seams there is another problem

$ docker-compose -p myproject ps
...
DEBUG:paramiko.transport:Secsh channel 0 opened.
DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
DEBUG:urllib3.connectionpool:ssh://localhost:None "GET /v1.22/networks/monitor_default HTTP/1.1" 200 None
DEBUG:paramiko.transport:[chan 0] EOF sent (0)
DEBUG:paramiko.transport:Secsh channel 1 opened.
DEBUG:paramiko.transport:[chan 1] Sesch channel 1 request ok
DEBUG:urllib3.connectionpool:ssh://localhost:None "GET /v1.22/containers/json?limit=-1&all=1&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dmyproject%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 None
DEBUG:paramiko.transport:[chan 1] EOF sent (1)
DEBUG:paramiko.transport:Secsh channel 2 opened.
DEBUG:paramiko.transport:[chan 2] Sesch channel 2 request ok
DEBUG:urllib3.connectionpool:ssh://localhost:None "GET /v1.22/containers/6ef0797c59e7ab988f20655047fc5889da14a875aedbe70951b34897be7f7dcf/json HTTP/1.1" 200 None
DEBUG:paramiko.transport:[chan 2] EOF sent (2)
DEBUG:paramiko.transport:Secsh channel 3 opened.
DEBUG:paramiko.transport:[chan 3] Sesch channel 3 request ok
...
DEBUG:paramiko.transport:[chan 7] EOF sent (7)
DEBUG:paramiko.transport:Secsh channel 8 opened.
DEBUG:paramiko.transport:[chan 8] Sesch channel 8 request ok
DEBUG:urllib3.connectionpool:ssh://localhost:None "GET /v1.22/containers/512456a8992c0e589af59280460a1c5e0032fb9f9befa31319ebe3f1feb9314c/json HTTP/1.1" 200 None
DEBUG:paramiko.transport:[chan 8] EOF sent (8)
DEBUG:paramiko.transport:Secsh channel 9 opened.
DEBUG:paramiko.transport:[chan 9] Sesch channel 9 request ok
DEBUG:urllib3.connectionpool:ssh://localhost:None "GET /v1.22/containers/aec33ad22a01bf232817e090cc0d147203ec24afa4dab484392dd4fc67e1201c/json HTTP/1.1" 200 None
DEBUG:paramiko.transport:[chan 9] EOF sent (9)
DEBUG:paramiko.transport:Secsh channel 10 opened.
DEBUG:paramiko.transport:[chan 10] Sesch channel 10 request ok
DEBUG:urllib3.connectionpool:ssh://localhost:None "GET /v1.22/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dmyproject%22%2C+%22com.docker.compose.oneoff%3DTrue%22%5D%7D HTTP/1.1" 200 3
DEBUG:paramiko.transport:[chan 10] EOF sent (10)
ERROR:compose.cli.errors:Couldn't connect to Docker daemon at http+docker://ssh - is it running?

If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
ERROR: Couldn't connect to Docker daemon at http+docker://ssh - is it running?

As you can see it list with "oneoff=False" and get each container.
Ans then is retake the container list with "oneoff=True" and fails.

[jelhan]

@ulyssessouza Are you sure the original issue is solved? Running latest versions of docker-compose (1.24.0) and docker (18.09.3) I'm still facing a paramiko.ssh_exception.ChannelException: (1, 'Administratively prohibited') issue. Increasing maxSession option of sshd fixes this issue. This was suggested as a work-a-round by @chris-crone in second post of this thread.

I could confirm that the AttributeError: 'NoneType' object has no attribute 'open_session' reported by @Vmadmax in forth post of this thread seems to be fixed.

[ssbarnea]

Today I faced the same bug and I can confirm that Adding MaxSessions 500 fixed the issue.

Still, because default is only 10 there is a huge chance that this will be encountered by most users trying to use the removing interface.

We need to verify this and give a meaning error to the user as we are talking about a default system configuration that will trigger this in 9/10 cases and not some corner-case scalability issue.

[derfabianpeter]

Had this bug, too. Since not everybody is able (or wants) to configure MaxSessions, the issue can be bypassed:

docker-compose -f docker-compose.yml -p $PROJECT config | ssh $USER@$HOST docker-compose -f - -p $PROJECT up --remove-orphans -d

Still assumes you have docker-compose available on both ends. Using this in CI.

[nfk]

Hello, same issue on my side with ubuntu 16.04, docker-compose 15.1 and docker 19.03.5. Updating the sshd configuration is required.

[woto]

Today I faced the same bug and I can confirm that Adding MaxSessions 500 fixed the issue.

Still, because default is only 10 there is a huge chance that this will be encountered by most users trying to use the removing interface.

We need to verify this and give a meaning error to the user as we are talking about a default system configuration that will trigger this in 9/10 cases and not some corner-case scalability issue.

It helps me. Thanks!

[amovfx]

yea 500 seemed to do the trick.

[nick-brady]

Just ran into this issue with a fairly standard compose configuration. I think the issue should be reopened. Setting MaxSessions in /etc/ssh/sshd_config on my remote host to

MaxSessions 500

followed by a

sudo service ssh restart

did fix the issue for me. Running on ubuntu bionic

[trevorlloydelliott]

Why is this closed? Original issue is not solved. Still encountering this with Docker 19.03.8.

[davetapley]

Just to call out: This issue will probably start getting more traffic now --compose has been added in 1.26.0 (in #7202), because that's how I ended up here.

---

@aiordache it might be worth updating your blog post to mention this issue (and the MaxSession workaround). Thanks for that post by the way, it was very helpful! 🙏

[andoks]

Still an issue with 1.26.2 on linux seems like

docker-compose version
docker-compose version 1.26.2, build eefe0d31
docker-py version: 4.2.2
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

[xmontero]

Question is... any solution "client side"? What if we can't change the sshd server to accept more connections and what we want to do is reduce the connections from the client controller?

I get the problem when doing docker-compose up -d with something as simple as 1 instance of a caddy:2 server and 4 instances of strm/helloworld-http just to play with an initial reverse proxy. Nothing complex.

$ docker-compose version
docker-compose version 1.27.0, build 980ec85b
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

[andoks]

It seems like this issue should have been fixed by https://github.com/docker/docker-py/pull/2275 which was merged early 2019, but weirdly the commits in the MR does not show up as part of any branches... Maybe my github-fu is not up to snuff, but if it is the case that the changes never hit the production branch, this issue and the issues that the MR was initially submitted to fix should probably be re-opened.