Compose: Compose on macOS, docker host on windows

[lox]

I've been working with https://github.com/StefanScherer/windows-docker-machine to be able to run windows docker containers from macOS. I'm having trouble mapping paths through. This works:

docker run --rm -it -v C:\\Users\\lachlan\\.ssh:C:\\ssh -w C:\\ssh microsoft/windowsservercore powershell

When I try and translate that to docker-compose.yml, e.g:

version: "3.2"

services:
  test:
    image: microsoft/windowsservercore
    volumes:
      - C:\\Users\\lachlan\\.ssh:C:\\ssh

And I get this error:
ERROR: Volume C:\\Users\\lachlan\\.ssh:C:\\ssh has incorrect format, should be external:internal[:mode]

Any ideas?

I'm running:

docker-compose version 1.16.1, build 6d1ac21 (macOS 10.12)

docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.30 (downgraded from 1.32)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:09 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.2-ee-5
 API version:  1.30 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   508bb92
 Built:        Thu Nov  2 00:54:07 2017
 OS/Arch:      windows/amd64
 Experimental: false

docker-info:

Containers: 9
 Running: 0
 Paused: 0
 Stopped: 9
Images: 17
Server Version: 17.06.2-ee-5
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.1770.amd64fre.rs1_release.170917-1700)
Operating System: Windows Server 2016 Standard Evaluation
OSType: windows
Architecture: x86_64
CPUs: 2
Total Memory: 2GiB
Name: vagrant-2016
ID: XJMU:DGZW:RZSC:CU2Q:YKGQ:5BUM:P726:2TTA:TQN3:EWHD:DNXB:5PMU
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[shin-]

Unfortunately we don't currently support this use-case (UNIX host to Windows containers), but I'm aware this is an issue. The changes I'm making in #5363 may help.

[StefanScherer]

I tried #5363 with this yml

version: "3.2"

services:
  test:
    image: microsoft/windowsservercore
    volumes:
      - C:\\Users\\stefan\\.ssh:C:\\ssh

networks:
  default:
    external:
      name: nat

but it shows this error:

Creating composeissue5371_test_1 ... error

ERROR: for composeissue5371_test_1  Cannot create container for service test: invalid volume specification: 'C:\\Users\\stefan\\.ssh:C:\\ssh:rw'

ERROR: for test  Cannot create container for service test: invalid volume specification: 'C:\\Users\\stefan\\.ssh:C:\\ssh:rw'
ERROR: Encountered errors while bringing up the project.

Then I tried slashes

version: "3.2"

services:
  test:
    image: microsoft/windowsservercore
    volumes:
      - C:/Users/stefan/.ssh:C:/ssh

networks:
  default:
    external:
      name: nat

but another error occurs with named volumes

{u'source': u'C:/Users/stefan/.ssh', u'type': u'volume', u'target': u'C:/ssh'}
ERROR: Named volume "{u'source': u'C:/Users/stefan/.ssh', u'type': u'volume', u'target': u'C:/ssh'}" is used in service "test" but no declaration was found in the volumes section.

Removing drive letter also does not help:

version: "3.2"

services:
  test:
    image: microsoft/windowsservercore
    volumes:
      - /Users/stefan/.ssh:/ssh

networks:
  default:
    external:
      name: nat

Creating composeissue5371_test_1 ... error

ERROR: for composeissue5371_test_1  Cannot create container for service test: invalid volume specification: '/Users/stefan/.ssh:/ssh:rw'

ERROR: for test  Cannot create container for service test: invalid volume specification: '/Users/stefan/.ssh:/ssh:rw'
ERROR: Encountered errors while bringing up the project.

[shin-]

@StefanScherer Yeah, that PR is heavy WIP still, absolutely not in a usable state. I'll ping you once it gets there :)

[StefanScherer]

@shin- Thanks, that would be fine.

Just drafted a quick Dockerfile.windows to compile docker-compose.exe from my Mac in a Windows container.

FROM python:2.7.14-windowsservercore

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

ENV GIT_VERSION 2.14.2
ENV GIT_DOWNLOAD_URL https://github.com/git-for-windows/git/releases/download/v${GIT_VERSION}.windows.1/MinGit-${GIT_VERSION}-64-bit.zip
ENV GIT_SHA256 9638733b8d749c43d59c34a714d582b2352356ee7d13c4acf919c18f307387f5

RUN Invoke-WebRequest -UseBasicParsing $env:GIT_DOWNLOAD_URL -OutFile git.zip; \
    if ((Get-FileHash git.zip -Algorithm sha256).Hash -ne $env:GIT_SHA256) {exit 1} ; \
    Expand-Archive git.zip -DestinationPath C:\git; \
    Remove-Item git.zip; \
    Write-Host 'Updating PATH ...'; \
    $env:PATH = 'C:\git\cmd;C:\git\mingw64\bin;C:\git\usr\bin;' + $env:PATH; \
    Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Environment\' -Name Path -Value $env:PATH

COPY . /code
WORKDIR /code
RUN pip install tox==2.1.1 virtualenv==13.1.2
RUN .\\script\\build\\windows.ps1

But have to copy the binary from the image back to my Mac with these commands so I can use it in a Win2016 VM.

$ docker build -t compose .
$ docker create compose
$ docker cp $(docker ps -ql):code/dist/docker-compose-Windows-x86_64.exe .

[shin-]

@StefanScherer PR should be more stable now - can you give it another shot? Thank you!

[StefanScherer]

@shin- Yeah, will fetch the PR again and build compose again. Would be much easier if we could use CircleCI for this repo to have artifacts (all cross compiled compose bins) for each PR/Git commit like we have in docker/cli repo. That saved me a lot of time to test docker manifest command 😅

[lox]

Out of interest, why can't CircleCI be used for this repo?

[shin-]

@lox @StefanScherer Is this still an issue with 1.18.0?

[lox]

Yup, I believe so. I will double check.

[StefanScherer]

@shin- With 1.18.0 I see this error

$ cat docker-compose.yml 

version: "3.2"

services:
  test:
    image: microsoft/windowsservercore
    volumes:
      - C:\\Users\\stefan\\.ssh:C:\\ssh

networks:
  default:
    external:
      name: nat



md5-cc35205805995ed665f093d6fd48b517



~/code/dockerfiles-windows/compose-issue-5371 on master*
$ docker-compose up
ERROR: Volume C:\\Users\\stefan\\.ssh:C:\\ssh has incorrect format, should be external:internal[:mode]

[shin-]

@StefanScherer Ok, I'll take another look at it then. Any luck with the extended syntax? i.e.

volumes:
  - type: bind
    source: C:\\Users\\stefan\\.ssh
    target: C:\\ssh

[StefanScherer]

@shin- With the extended syntax I get

$ docker-compose up
Creating composeissue5371_test_1 ... error

ERROR: for composeissue5371_test_1  Cannot create container for service test: invalid volume spec "C:\\\\ssh": invalid volume specification: 'C:\\ssh'

ERROR: for test  Cannot create container for service test: invalid volume spec "C:\\\\ssh": invalid volume specification: 'C:\\ssh'
ERROR: Encountered errors while bringing up the project.

Then I tried target: C:/ssh with unixish windows path, but then I get another error

Creating composeissue5371_test_1 ... error

ERROR: for composeissue5371_test_1  Cannot create container for service test: invalid bind mount spec "C:\\\\Users\\\\stefan\\\\.ssh:C:/ssh:rw": invalid volume specification: 'C:\\Users\\stefan\\.ssh:C:\ssh:rw'

ERROR: for test  Cannot create container for service test: invalid bind mount spec "C:\\\\Users\\\\stefan\\\\.ssh:C:/ssh:rw": invalid volume specification: 'C:\\Users\\stefan\\.ssh:C:\ssh:rw'

[StefanScherer]

Thanks @shin- for the tip using single backslashes. It works with docker-compose 1.18.0 and single backslashes and the extended syntax: 🎉

version: "3.2"

services:
  test:
    image: microsoft/windowsservercore
    volumes:
      - type: bind
        source: C:\Users\stefan\.ssh
        target: C:\ssh

networks:
  default:
    external:
      name: nat

$ docker-compose up
Starting composeissue5371_test_1 ... done
Attaching to composeissue5371_test_1
test_1  | Microsoft Windows [Version 10.0.14393]
test_1  | (c) 2016 Microsoft Corporation. All rights reserved.
test_1  | 
test_1  | C:\>
composeissue5371_test_1 exited with code 0
~/code/dockerfiles-windows/compose-issue-5371 on master*

[lox]

Nice one. Whilst . doesn't work, this gets it done!

    volumes:
      - type: bind
        source: C:${PWD}
        target: C:/gopath/src/github.com/buildkite/agent

[lox]

But of course Docker on Windows uses symlinks for it's binds. 😭