Compose: Failed to execute script docker-compose when doing docker-compose up on SSH context

[bibekg]

Description of the issue

Saw this post detailing how to use contexts with the upcoming docker-compose 1.26.0-rc3 release and tried to give it a shot. I then installed that pre-release version on both my local machine which has the application source code and the new EC2 machine that I want to deploy to and created a context called rolocha-staging which connects to the docker engine in the EC2 instance. Then I tried to deploy my docker-compose app to it using either

docker context use rolocha-staging
docker-compose --env-file .env.staging -f docker-compose.yml -f docker-compose.staging.yml up -d

or

docker-compose --context rolocha-staging --env-file .env.staging -f docker-compose.yml -f docker-compose.staging.yml up -d

but both approaches are resulting in the error/stacktrace included below.

If I execute in the default context (i.e. use --context default argument or docker context use default command), everything works smoothly.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.26.0-rc3, build 46118bc5

Output of docker version

Docker version 19.03.8, build afacb8b

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

networks:
  app-network:
    driver: bridge
services:
  client:
    build:
      context: /Users/bibek/Developer/RoloCha/client
    command: npx serve -s build -l 4000
    container_name: rolocha-client
    environment:
      APP_CLIENT_PORT: '4000'
      APP_SERVER_PORT: '3000'
      MONGO_DB_NAME: rolocha-dev
      MONGO_HOSTNAME: db
      MONGO_PASSWORD: [redacted]
      MONGO_PORT: '27017'
      MONGO_USERNAME: admin
      NODE_ENV: development
    expose:
    - '4000'
    image: roloch4/client-staging:latest
    networks:
      app-network: {}
    ports:
    - 4000:4000/tcp
    restart: always
  db:
    container_name: db
    environment:
      APP_CLIENT_PORT: '4000'
      APP_SERVER_PORT: '3000'
      MONGO_DB_NAME: rolocha-dev
      MONGO_HOSTNAME: db
      MONGO_INITDB_ROOT_PASSWORD: ''
      MONGO_INITDB_ROOT_USERNAME: ''
      MONGO_PASSWORD: [redacted]
      MONGO_PORT: '27017'
      MONGO_USERNAME: admin
      NODE_ENV: development
    image: mongo
    networks:
      app-network: {}
    ports:
    - 27017:27017/tcp
    volumes:
    - dbdata:/data/db:rw
  nginx:
    container_name: nginx
    image: nginx:latest
    networks:
      app-network: {}
    ports:
    - 80:80/tcp
    - 443:443/tcp
    restart: always
    volumes:
    - /Users/bibek/Developer/RoloCha/nginx-conf/nginx.conf:/etc/nginx/nginx.conf:rw
  server:
    build:
      context: /Users/bibek/Developer/RoloCha/server
    command: yarn start
    container_name: rolocha-server
    environment:
      APP_CLIENT_PORT: '4000'
      APP_SERVER_PORT: '3000'
      MONGO_DB_NAME: rolocha-dev
      MONGO_HOSTNAME: db
      MONGO_PASSWORD: [redacted]
      MONGO_PORT: '27017'
      MONGO_USERNAME: admin
      NODE_ENV: development
    expose:
    - '3000'
    image: roloch4/server-staging:latest
    networks:
      app-network: {}
    ports:
    - 3000:3000/tcp
    restart: always
version: '3.0'
volumes:
  dbdata: {}
  node_modules_c: {}
  node_modules_s: {}

Steps to reproduce the issue

1. Create a remote context connecting to an EC2 instance via SSH using docker context create
2. Switch to that context using docker context use <context-name>
3. Try to deploy the local docker-compose app to the remote context using docker-compose up

Observed result

Error message above

Expected result

No error message, starts up the docker-compose services in remote machine

Stacktrace / full error message

Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 72, in main
  File "compose/cli/main.py", line 125, in perform_command
  File "compose/cli/command.py", line 54, in project_from_options
  File "compose/cli/docker_client.py", line 41, in load_context
  File "site-packages/docker/context/api.py", line 97, in get_context
  File "site-packages/docker/context/context.py", line 61, in load_context
  File "site-packages/docker/context/context.py", line 87, in _load_meta
KeyError: 'StackOrchestrator'
[92007] Failed to execute script docker-compose
make: *** [stageup] Error 255

Additional information

Running macOS Catalina 10.15.4

Installed docker-compose v1.2.6-rc3 on both local machine and EC2 instance using

curl -L https://github.com/docker/compose/releases/download/1.26.0-rc3/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

[icarus42]

I was able to fix this issue by adding the parameter --default-stack-orchestrator swarm when creating the context:

docker context create NAME --description "DESCRIPTION" --docker "host=ssh://user@example" --default-stack-orchestrator swarm

After that, I had the issue that compose could not access my encrypted SSH private key. I just decrypted it for now...

Worked for:

• docker compose: 1.26.0-rc3, build 46118bc5
• docker ce: 19.03.8

[kylelemons]

I ran into the same issue:

$ docker-compose --context=remote logs letsencrypt
Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose\cli\main.py", line 72, in main
  File "compose\cli\main.py", line 125, in perform_command
  File "compose\cli\command.py", line 54, in project_from_options
  File "compose\cli\docker_client.py", line 41, in load_context
  File "site-packages\docker\context\api.py", line 97, in get_context
  File "site-packages\docker\context\context.py", line 61, in load_context
  File "site-packages\docker\context\context.py", line 87, in _load_meta
KeyError: 'StackOrchestrator'
[29084] Failed to execute script docker-compose

The same fix worked for me:

$ docker context update remote --default-stack-orchestrator swarm

---

Local (Windows 10):

$ docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:23:10 2020
 OS/Arch:           windows/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.5
  Git commit:       369ce74
  Built:            Wed May  6 07:28:37 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.2
  GitCommit:        ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
 runc:
  Version:          1.0.0-rc10
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683b971d9c3ef73f284f176672c44b448662

Remote (gce instance running cos, latest stable):

$ docker --host=localhost:22122 version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:23:10 2020
 OS/Arch:           windows/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.5
  Git commit:       369ce74
  Built:            Wed May  6 07:28:37 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.2
  GitCommit:        ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
 runc:
  Version:          1.0.0-rc10
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683b971d9c3ef73f284f176672c44b448662

Compose:

$ docker-compose version
docker-compose version 1.26.0-rc4, build d279b7a8
docker-py version: 4.2.0
CPython version: 3.7.4
OpenSSL version: OpenSSL 1.1.1c  28 May 2019     

[azhang]

The above post by @kylelemons fixed some stuff for me, but I get the following issue with and without --context set for docker-compose.

~/s/b/d/nginx-certbot git:master ❯❯❯ docker-compose-pre --context btapi-dev up                                                              ⏎ ⬆ ✭ ✚ ✱ ◼
Starting nginx-certbot_certbot_1 ... 
Starting nginx-certbot_nginx_1   ... 
ERROR: Secsh channel 12 open FAILED: open failed: Connect failed
Starting nginx-certbot_certbot_1 ... done
ERROR: for nginx-certbot_nginx_1  ChannelException(2, 'Connect failed')

ERROR: for nginx  ChannelException(2, 'Connect failed')
Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 72, in main
  File "compose/cli/main.py", line 128, in perform_command
  File "compose/cli/main.py", line 1078, in up
  File "compose/cli/main.py", line 1074, in up
  File "compose/project.py", line 576, in up
  File "compose/parallel.py", line 112, in parallel_execute
  File "compose/parallel.py", line 210, in producer
  File "compose/project.py", line 562, in do
  File "compose/service.py", line 569, in execute_convergence_plan
  File "compose/service.py", line 511, in _execute_convergence_start
  File "compose/parallel.py", line 112, in parallel_execute
  File "compose/parallel.py", line 210, in producer
  File "compose/service.py", line 509, in <lambda>
  File "compose/service.py", line 620, in start_container_if_stopped
  File "compose/container.py", line 215, in attach_log_stream
  File "compose/container.py", line 307, in attach
  File "site-packages/docker/utils/decorators.py", line 19, in wrapped
  File "site-packages/docker/api/container.py", line 60, in attach
  File "site-packages/docker/utils/decorators.py", line 46, in inner
  File "site-packages/docker/api/client.py", line 226, in _post
  File "site-packages/requests/sessions.py", line 581, in post
  File "site-packages/requests/sessions.py", line 533, in request
  File "site-packages/requests/sessions.py", line 646, in send
  File "site-packages/requests/adapters.py", line 449, in send
  File "site-packages/urllib3/connectionpool.py", line 677, in urlopen
  File "site-packages/urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1252, in request
  File "http/client.py", line 1298, in _send_request
  File "http/client.py", line 1247, in endheaders
  File "http/client.py", line 1026, in _send_output
  File "http/client.py", line 966, in send
  File "site-packages/docker/transport/sshconn.py", line 32, in connect
  File "site-packages/paramiko/transport.py", line 879, in open_session
  File "site-packages/paramiko/transport.py", line 1017, in open_channel
paramiko.ssh_exception.ChannelException: ChannelException(2, 'Connect failed')
[24485] Failed to execute script docker-compose

However I ran the command again and it worked. My ssh connection is very stable otherwise.

[ruanbekker]

Getting the same as @azhang , im using:

$ /tmp/docker-compose -v
docker-compose version 1.26.0-rc4, build d279b7a8

[hazcod]

Having the same issue with regular docker (non swarm), DOCKER_HOST=ssh://host.

+ docker-compose -p xxx -f compose/network.yml -f compose/db.yml -f compose/php.yml -f compose/web.yml -f compose/download.yml -f compose/mq.yml -f compose/slave.yml -f compose/processor.yml -f compose/stages/prod/prod.yml up -d
659
Some services (db, download, mq, php, processor, slave, web) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
660
Creating network "xxx" with driver "bridge"
661
Creating volume "xxx" with default driver
662
Creating volume "xxx-data" with default driver
663
db is up-to-date
664
Creating download ... 
665
Creating mq       ... 
666
Secsh channel 49 open FAILED: open failed: Connect failed
667

668
ERROR: for download  ChannelException(2, 'Connect failed')
669
Secsh channel 51 open FAILED: open failed: Connect failed
670
[2838] Failed to execute script docker-compose
671

672
ERROR: for mq  ChannelException(2, 'Connect failed')
673

674
ERROR: for download  ChannelException(2, 'Connect failed')
675

676
ERROR: for mq  ChannelException(2, 'Connect failed')`

[hazcod]

logs_93.zip
Verbose logs attached. (docker-compose --verbose up)
Also happens with compose 26.0.