Compose: Secrets not bind mounted in a service

[bbinet]

With the following compose file, no secrets are bind mounted in the saltminion service whereas the secrets of the saltmaster service are correctly bind mounted:

version: '3.1'
services:
  saltmaster:
    image: bbinet/salt-master:latest
    hostname: hlmaster
    environment:
      PRE_ACCEPT_MINIONS: hlminion
      PRE_CREATE_USERS: heliosalt
    ports:
    - 10.0.0.1:4433:443
    volumes:
    - ./saltmaster/etc_salt_master.d:/etc/salt/master.d
    - saltmaster-states:/srv/setupify/states:ro
    - saltmaster-reclass:/srv/setupify/reclass:ro
    secrets:
    - heliosalt.password
    - master.pem
    - master.pub
    - source: minion.pub
      target: hlminion.pub
  saltminion:
    image: bbinet/salt-minion:jessie
    hostname: hlminion
    cap_add:
    - SYS_ADMIN
    environment:
      TERM: xterm
    secrets:
    - minion.pem
    - minion.pub
    - master.pub
    volumes:
    - /sys/fs/cgroup:/sys/fs/cgroup
    - ./saltminion/etc_salt_minion.d:/etc/salt/minion.d
volumes:
  saltmaster-reclass:
    driver_opts:
      device: /home/bruno/dev/setupify_reclass/reclass
      o: bind
      type: none
  saltmaster-states:
    driver_opts:
      device: /home/bruno/dev/setupify_reclass/states
      o: bind
      type: none
secrets:
  heliosalt.password:
    file: ./secrets/heliosalt.password
  master.pem:
    file: ./secrets/master.pem
  master.pub:
    file: ./secrets/master.pub
  minion.pem:
    file: ./secrets/minion.pem
  minion.pub:
    file: ./secrets/minion.pub

$ tree
.
├── docker-compose.yml
├── saltmaster
│   └── etc_salt_master.d
│       └── helioslite.conf
├── saltminion
│   └── etc_salt_minion.d
│       └── helioslite.conf
└── secrets
    ├── heliosalt.password
    ├── master.pem
    ├── master.pub
    ├── minion.pem
    └── minion.pub



md5-2bb69aa030d76d1342d66ffddea74293



$ docker exec -ti saltstack_saltminion_1 ls /run/secrets
ls: cannot access /run/secrets: No such file or directory
$ docker exec -ti saltstack_saltmaster_1 ls /run/secrets
heliosalt.password  hlminion.pub  master.pem  master.pub



md5-229a2d34c41cbc29529b556f9bf8218d



$ docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:04:27 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:04:27 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ docker-compose version
docker-compose version 1.13.0, build 1719ceb
docker-py version: 2.3.0
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1t  3 May 2016

Please ask if you want me to provide the result of docker inspect saltstack_saltminion_1 or any other information to help debug this issue.

[bbinet]

Note that bind mounted secrets has been introduced in docker-compose in PR #4368.

[shin-]

@bbinet Thank you for the detailed report. Do you mind sharing the output of the docker-compose config command?

[bbinet]

@shin- sure:

networks: {}
secrets:
  heliosalt.password:
    file: /home/bruno/dev/setupify_reclass/compose/saltstack/secrets/heliosalt.password
  master.pem:
    file: /home/bruno/dev/setupify_reclass/compose/saltstack/secrets/master.pem
  master.pub:
    file: /home/bruno/dev/setupify_reclass/compose/saltstack/secrets/master.pub
  minion.pem:
    file: /home/bruno/dev/setupify_reclass/compose/saltstack/secrets/minion.pem
  minion.pub:
    file: /home/bruno/dev/setupify_reclass/compose/saltstack/secrets/minion.pub
services:
  saltmaster:
    environment:
      PRE_ACCEPT_MINIONS: hlminion
      PRE_CREATE_USERS: heliosalt
    hostname: hlmaster
    image: bbinet/salt-master:latest
    ports:
    - 10.0.0.1:4433:443/tcp
    secrets:
    - source: heliosalt.password
    - source: master.pem
    - source: master.pub
    - source: minion.pub
      target: hlminion.pub
    volumes:
    - /home/bruno/dev/setupify_reclass/compose/saltstack/saltmaster/etc_salt_master.d:/etc/salt/master.d:rw
    - saltmaster-states:/srv/setupify/states:ro
    - saltmaster-reclass:/srv/setupify/reclass:ro
  saltminion:
    cap_add:
    - SYS_ADMIN
    environment:
      TERM: xterm
    hostname: hlminion
    image: bbinet/salt-minion:jessie
    secrets:
    - source: minion.pem
    - source: minion.pub
    - source: master.pub
    volumes:
    - /sys/fs/cgroup:/sys/fs/cgroup:rw
    - /home/bruno/dev/setupify_reclass/compose/saltstack/saltminion/etc_salt_minion.d:/etc/salt/minion.d:rw
version: '3.1'
volumes:
  saltmaster-reclass:
    driver_opts:
      device: /home/bruno/dev/setupify_reclass/reclass
      o: bind
      type: none
  saltmaster-states:
    driver_opts:
      device: /home/bruno/dev/setupify_reclass/states
      o: bind
      type: none

[shin-]

@bbinet Are you still encountering this issue with Compose 1.14.0? My attempts at reproducing have been unsuccessful.

[bbinet]

I've just upgraded to docker-compose 1.14.0 and docker 17.06.0-ce, and the problem is the same.

$ docker-compose version
docker-compose version 1.14.0, build c7bdf9e
docker-py version: 2.4.2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1t  3 May 2016

$ docker version
Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:20:04 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:18:59 2017
 OS/Arch:      linux/amd64
 Experimental: false



md5-2bb69aa030d76d1342d66ffddea74293



$ docker info
Containers: 3
 Running: 2
 Paused: 0
 Stopped: 1
Images: 94
Server Version: 17.06.0-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Kernel Version: 4.3.0-0.bpo.1-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.68GiB
Name: hl-wrk-1-dev
ID: Q4AY:KFKP:ISPD:P5HM:I3A4:HQ24:KBJP:CRBH:W6KO:X7BJ:7HBX:FK4Z
Docker Root Dir: /media/docker
Debug Mode (client): false
Debug Mode (server): false
Username: helioslite
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support

[bbinet]

@shin- You mean that when you take the same docker compose file, same setup, it does work for you?

[shin-]

@bbinet yes indeed. I can see files mounted accordingly in both containers.

[rhencke]

I can reproduce this issue. (Well... I think it's this issue)

It seems that the 'file' argument's value is _always ignored_. Instead, if the 'file' argument is present, it will look for a file name with the same name as the secret.

For example, this is OK, assuming the presence of a 'hello' file in the same directory as docker-compose.yaml, this runs successfully. This is using docker-compose up.

version: '3.3'
secrets:
  hello:
    file: ./it/doesnt/even/matter/what/you/put
services:
  cathello:
    image: alpine
    command: "cat /run/secrets/hello"
    secrets:
      - hello

This is docker-compose version 1.16.1, build 6d1ac21, Docker version 17.09.0-ce, build afdb6d4

[shin-]

@rhencke That doesn't seem to be related. Can you open a new issue with more details and a repro case? As far as I can tell, the file field is taken into account in my test apps, but maybe there's a weird corner case I'm not hitting?

[shin-]

@bbinet Let me know if this is still an issue for you. As I mentioned, this same Compose file mounts secrets in both services when I test it on my setup, so I don't think it's a bug in Compose.

[bbinet]

@shin- yes this is still an issue for me

[bbinet]

I had some time to continue investigation, and I discovered that this problem occurs only when systemd is running in the container.
But I have no clue why it prevents docker from mounting the secrets in /run/secrets...
Any idea?

[bbinet]

I've finally managed to make it work by mouting /run as tmpfs with the following addition to my docker-compose.yml file:

tmpfs:
  - /run

So I'm closing this issue.

[jmoraleda]

The work around of adding

tmpfs:
  - /run

worked for me, but I still think this is a bug, even if there is a work around. Also, when mounting only one secret, this is not required.

[rmarren1]

I am also getting this error, and the tmpfs: /run workaround is not working.

Here's my dockerfile

version: '3.1'
services:

  database:
    container_name: mongo
    image: mongo
    volumes:
        - datavolume:/data/db
    ports:
      - "27017:27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: "root"
      MONGO_INITDB_ROOT_PASSWORD_FILE: /run/secrets/mongo_password
    secrets:
      - mongo_password

  api:
    container_name: api
    image: username/api
    build: api
    ports:
      - "8080:8080"
    links:
      - database
    depends_on:
      - database
    environment:
      DEBUG: 1
      DOCKER: 1
    secrets:
      - mongo_password


  app:
    container_name: app
    image: username/app
    build: app
    ports:
      - "8050:8050"
    links:
      - api
    depends_on:
      - api
    environment:
      DEBUG: 1
      DOCKER: 1
    secrets:
      - mongo_password

volumes:
    datavolume: {}

secrets:
  mongo_password:
    file: ./auth/mongo-password

docker-compose will silently ignore the secret file auth/mongo-password and default to making run/secrets/mongo_password an empty directory.

[aharpervc]

I hit this too. Here's a simplified docker compose file:

version: '3.1'
services:
  ub_test:
    image: ubuntu
    command: ls -la /run/secrets
    secrets:
      - host_github_ssh_key
secrets:
  host_github_ssh_key:
    file: ~/.ssh/id_rsa

[ningcui-refundlabs]

I hit this too. Here's a simplified docker compose file:

version: '3.1'
services:
  ub_test:
    image: ubuntu
    command: ls -la /run/secrets
    secrets:
      - host_github_ssh_key
secrets:
  host_github_ssh_key:
    file: ~/.ssh/id_rsa

same issue. Did you find a solution?

[aharpervc]

same issue. Did you find a solution?

Nope. I ended up not using docker compose secrets at all, and instead reconfigured my situation to copy in secrets during container build (but using multi stage builds to discard them again in the final image).

[ningcui-refundlabs]

same issue. Did you find a solution?

Nope. I ended up not using docker compose secrets at all, and instead reconfigured my situation to copy in secrets during container build (but using multi stage builds to discard them again in the final image).

Totally understood. I found a lot of articles and answers, but none of them works.

[ewilansky]

I saw the same behavior where /run/secrets doesn't get bind-mounted into the container when using Docker secrets in docker-compose. Upon further investigation, I could see that the base image (in my case the Sonarqube 7.5-community image has a /run folder in the image. There isn't a /run/secrets folder, but perhaps the presence of the the /run folder or a permission issue keeps /run/secrets from getting bind mounted? My solution was to use the docker secrets advanced syntax in Docker 17.06 and beyond to specify a custom location for the target bind mount folder. By using /usr/local/secrets as the target, I had no trouble projecting the secrets into the container:

sonarqube: 
    ...
    secrets:
       - source: sonarqube-passwd
         target: /usr/local/secrets/sonarqube-passwd
       - source: sonarqube-user
         target: /usr/local/secrets/sonarqube-user
    ...

and at the root of the docker-compose:

secrets:
  sonarqube-user:
    file: ./secrets/sonarqube_usr.txt
  sonarqube-passwd:
    file: ./secrets/sonarqube_password.txt>

In the container:

[CatEars]

I had this problem but I managed to fix it by removing the containers and restarting them with compose.

Here is a minimal docker-compose.yml to produce the problem I had

version: '3.4'

secrets:
  A:
    file: ./secret.txt
services:
  nginx:
    image: nginx

    # Run docker-compose up and then uncomment me and run again!
    # secrets:
    #   - A

To remove the container simply find it with docker container ls -a, copy its ID and then remove with docker rm $THE_ID

[lukens]

I was having the same problem, and tipped off by @CatEars suggestion of deletion, I deleted all containers and images. It then worked. Trying to be a bit more scientific, I think I narrowed it down to deleting the image that fixes it (though, this makes no sense to me).

[hckhanh]

I am having the same problem with these guys, I am currently using Docker 3.7 but I still get stuck

version: "3.7"
services:
  web:
#    image: node
#    entrypoint: sudo ls -la /usr/local/secrets
    build: .
    ports:
      - "8000:8000"
#    depends_on:
#      - db
#      - redis
    environment:
      PRIVATE_KEY_PATH: /usr/local/secrets/private_key
    secrets:
      - private_key
  redis:
    image: redis
  db:
    image: mysql
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: test
secrets:
  private_key:
    external: true

[robertsimoes]

Do we know if this issue has been resolved?