Compose: Epic: Windows support

[bfirsh]

This is a meta-issue for keeping track of all the things we're doing around Windows support for Compose.

Various things that are part of this:

• Building a binary on Windows. PyInstaller, the method we currently use to build binaries for Linux and OS X, also works with Windows. In theory this should just work. It would be useful to have build scripts etc to automate this.
• Run Compose inside a container. This may be be easier than building a native binary for Windows. https://github.com/docker/compose/issues/169 https://github.com/docker/compose/pull/1806
• Making Compose Windows compatible. I expect this will be mostly making sure paths work and getting the TTYs to work.
• Volume sharing. When running Compose on OS X, it expects /Users to be mounted in the right place so volumes work. I wonder if we need something added to Machine/boot2docker to make volumes work as expected on Windows. (The Compose quick start guide has a demonstration of how volumes are used to mount code inside a container.)
• Run CI tests on Windows. We don't all test on Windows, so we'd have to be careful about regressions. https://github.com/docker/compose/issues/886

Steps:

• [x] Investigate whether binaries will build cleanly on Windows https://github.com/docker/compose/issues/1051
• [x] Investigate whether running Compose inside a container will be easier than running it natively on Windows

Related issues:

• https://github.com/docker/compose/issues/598

[mikehaertl]

So you talk about a native Windows version of docker-compose here?

As there isn't even a docker version for windows yet, I wonder if it wouldn't make more sense to focus on including docker-compose into the boot2docker VM for now:

https://github.com/boot2docker/boot2docker/issues/603

This is something that blocks many developers on Windows machines. And even though there are workarounds, I'd consider them hacks. Windows users should not have such a hard time to work with docker-compose.

[funkyfuture]

Using Linux containers on Windows. Yeah, that's epic.

[dduportal]

@mikehaertl The docker client is not a requirement for docker-compose (which embed its own python-written client). For example, we use https://github.com/rhuss/docker-maven-plugin in my companyy, which can contact our boot2docker, but running a JVM process in Windows.

So there are two main use cases :

• Running Compose inside boot2docker (at OS level or inside a container)
• Running compose from your Host OS

=> In the two cases, compiling and running is just some plumbing. The main goal will be making that usable.
The real tecnical problem is the data sharing.

[bfirsh]

@mikehaertl Windows support is being added to the Docker client. https://github.com/docker/docker/pull/9113 https://github.com/docker/docker/pull/10864

Happy to accept changes which make Compose work inside boot2docker, but I think the ideal situation would be for Docker to run on Windows natively.

[funkyfuture]

now seriously,
i see a lot of issues open for a long time.
from my experience, adding and maintaining Windows-support is a real time-eater and will constantly bring up issues.
I'm not opposing Windows-support at all, but i'd say that should be done when few issues are on the backlog. especially functional enhancements.

and on the other hand i'm still irritated what that whole Docker on Windows-thing is actually aiming at. just to claim cross-platform-support? save Microsoft from getting irrelevant on servers? (yep, that's ot, no need to discuss it here.)

[kuborgh-mspindelhirn]

We use docker partilly for development machine setup and Windows support would be really cool.

I don´t really think this is a supprise as docker-machine and docker ( windows client is buildable since the start of the year ) support windows. Is there any reason that docker-compose should not?

[atrauzzi]

Slightly related perhaps: https://windows.uservoice.com/forums/265757-windows-feature-suggestions/suggestions/6573649-full-posix-support

Would having proper POSIX support like on OSX help? As a community, maybe it's worthwhile to at least help drive that message?

[kskalvar]

Would a cygwin version be workable? I used cygwin on windows and it's awesome.

[tknerr]

@ahmetalpbalkan will you be involved in this, too? Being able to docker-compose on windows would be awesome!

[ahmetb]

@tknerr happy to help! We have quite some Windows porting knowledge we can transfer from docker, boot2docker-cli, docker-machine. :smile:

[objarni]

I'd be happy to test any docker-compose binary or workaround script in a Windows 7 environment as the tricky nature of installing it stopped me today. I tried these methods, both failed: 1) clone and build docker-image myself as per stackoverflow answer here: http://stackoverflow.com/a/29728993 2) "pip install docker-compose" (got Py2.7 installed on machine). I'm using boot2docker, installed by official Windows installer.

[tknerr]

:+1: same for me, would be happy to help testing docker-compose on windows too

[ereOn]

I don't know if it was considered, but please don't enforce the use of cygwin on Windows.

Also regarding the generation of binaries on Windows, wouldn't entry points be good-enough solution ?

[tknerr]

+1 for that

Docker client since 1.6.0 runs perfectly on windows without cygwin, and docker-compose should too imho

[kskalvar]

I could use the docker image to run docker-compose on windoze but for some reason it complains it can't find the *.yml file.....

[dacay]

+1

[inaffect-ag]

+1

[qxo]

+1

[stagas]

+1

[willylambert]

If it can help, I successfully install and run docker-compose with boot2docker under Windows through the following commands which use the python package manager pip:

wget -P /tmp/ http://www.tinycorelinux.net/5.x/x86/tcz/python.tcz && tce-load -i /tmp/python.tcz && rm -f /tmp/python.tcz
curl -LO -k https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py && sudo /usr/local/bin/python2.7 ez_setup.py && rm -f ez_setup.py
sudo /usr/local/bin/easy_install-2.7 pip
sudo /usr/local/bin/pip2.7 install -U docker-compose==1.2.0

The only drawback is the need to install again docker-compose after each restart of the boot2docker VM.

[willylambert]

I run these commands directly in the boot2docker VM - so once installed you need to log into the VM to access docker-compose.

[willylambert]

Yes, my project code is on my host, I use the VirtualBox shared folders functionality.

[secat]

+1 on this epic.

[secat]

+1 on this epic.

[pbearne]

+1

[edmorley]

In the hope this saves others many hours of trying to figure out workarounds in the meantime:

• There appear to be caching issues with the "docker-compose in a container" method (dduportal-dockerfiles/docker-compose#6) - which means that changes to your docker-compose.yml/Dockerfile are not picked up by Compose, so I'd recommend staying away from the alias docker-compose="docker run ..." technique.
• Using the Compose PyPI package natively on Windows should also be avoided for now, since:
  
  
  
  • it also uses the same "docker-compose in a container" method, so will hit the caching bug too.
  
  
  • when using native Windows Python it fails with ImportError: No module named fcntl (d11wtq/dockerpty#7).
  
  
  • when using MSYS2's python, it failed with SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:585). _(Edit: appears to be a bug with Python 2.7.9 and openssl v1.0.2, #1484 - downgrading to 1.0.1 using the package archive here fixes the problem.)_
  
  
  • at one point it failed to load a docker-compose.yaml which contained comments (#1524) - possibly due to an upstream PyYAML issue (http://pyyaml.org/ticket/204) - though I can't repro any more.
  
  

As such for now, I'd advise Windows users to stick to installing the docker-compose Python package inside boot2docker, rather than the alternatives [1].

This is pretty painless - and can be made to persist after boot2docker restarts - see:
https://github.com/boot2docker/boot2docker/issues/603#issuecomment-110033929

[1] unless you likewise have a fondness for yak shaving. All I wanted to do was fix one bug in another team's project, then ended up learning Docker/Docker Compose to try and overhaul their development environment - and now... :-)

[dduportal]

Hello ! As said here : https://github.com/boot2docker/boot2docker/issues/603#issuecomment-111816296 you have a resume on the things around inside boot2docker.

Windows part need python compilation/packaging natively to run remote (or statically go rewrite) + docker new volumes system well running to ensure filesystem coherency ,

[phazei]

Running it non-native is kind of annoying, sure it could be run in boot2docker, but clearly that's not good enough or it wouldn't have been necessary to create the native docker.exe client.

Since docker-compose is python, and it's not doing anything incredibly specific, it should just run in windows under python, but it relies on the fcntl package for what I presume is file locking of some sort. Wouldn't an alternative method for file locking simply fix the issue?
Seems they came across the same issue and came up with some windows-friendly locking with this here
Edit: better link
Could a similar implementation be used?

[matthewr6]

@phazei
It also requires termios (found out after you put a workaround in such as the mentioned, or this)

@edmorley
Would that method work with Kitematic Windows' alpha?

[wklm]

absolutely +1!

[bfirsh]

Updated the description with latest developments of running Compose in a container. https://github.com/docker/compose/pull/1806 Thanks @dnephin!

[ahmetb]

@bfirsh libcompose should be helping a great deal delivering Compose for Windows, right? I don't see the point of running it inside a container if we will have a Go binary for it.

[bfirsh]

@ahmetalpbalkan Anything powered by libcompose is still a long way out.

If there's anything we can get Compose working with Windows in a short-term, then that's a win. :)

[nathanleclaire]

FYI the Virtualbox Shared Folders hack happens in virtualbox machines created with docker-machine in Windows today. The C:\Users directory is mounted at /c/users/ in the boot2docker VM IIRC.

This is largely problematic with the Docker CLI / msysgit (because there's a "translation layer" which attempts to munge Unix-style paths to Windows paths in msysgit), but might be something Compose can work around since it is sending information directly to the Docker API.

[friism]

I don't think the virtualbox shared folder approach can be counted on. This is because the virtualbox mapping of filesystem semantics from Windows to Linux is too imperfect (I think the particular problem I found was that something in bundle install used symlinks for something, and that didn't map out to the windows shared folder correctly).

[koushikckm]

@willylambert This is not working on docker
curl -LO -k "https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py" && sudo /usr/local/bin/python2.7 ez_setup.py && rm -f ez_setup.py
says
rning: Failed to create the file ez_setup.py: Permission denied

[pug5]

I recently developed a nice series of steps for getting Docker Compose and Docker Machine working with my Windows 10 setup. Hope this helps!

http://brianpuglisi.com/how-to-get-docker-compose-working-on-windows-10/

[matthewr6]

Would it work w/Kitematic?

[dopry]

PR #1900 get's docker-compose working on windows. I've been using it tonight.

[sirinath]

+1

[alexandarp]

I would love this feature +1

[bill2004158]

+1

[christianhuening]

+1 Actually need this feature very much!

[dopry]

Feel free to

pip install --upgrade git+https://github.com/dopry/compose.git@windows

or better yet check out https://github.com/docker/libcompose the libcompose cli works on windows as well, and has equivalent functionality and is written in go like the rest of the docker tooling suite.

[aanand]

See https://github.com/docker/compose/pull/1957 for my progress on building a Windows binary.

[lifenautjoe]

+1

[koushikckm]

+1. cool stuff.
I work on docker on windows 8. It works well but after certain period it
runs out of memory and crashes.

On Sun, Sep 6, 2015 at 4:55 PM, Joel Hernández [email protected]
wrote:

+1

—
Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/1085#issuecomment-138073237.

[divyakumarjain]

+1

[wreiske]

+1

[shantanuthatte]

pip install git+git://github.com/docker/compose.git

Works for me on Windows 7.

[T1m1]

Same here! :+1:

[atrauzzi]

Awesome!

Any idea when this will make it into the standard Windows release of Docker Toolbox?

[skygeo]

Awesome! Works like a charm in windows 7 !

[christianhuening]

@atrauzzi: i would like to know the same

[divyakumarjain]

Works for me with Phython27, pip on Windows 8.1.

[jamespacileo]

Weird issues on Windows 10, but I think it's Vagrant and toolbox :)

Very excited to try this out! Thank you!! :smile:

[Duske]

Works for me with Python3.4, pip, docker toolbox 1.8.2c and Windows 8.1.
Awesome that I can start using docker-compose.yml :)

[atrauzzi]

What's the preferred way to get Python installed nowadays? I'd rather wait for toolbox to update, but if that isn't soon enough, I suppose I can get it set up.

[parhamdoustdar]

I personally get a "Touple out of range" whenever I run the command to
install docker-compose with pip.

On 10/2/2015 11:44 PM, Alexander Trauzzi wrote:

What's the preferred way to get Python installed nowadays? I'd rather
wait for toolbox to update, but if that isn't soon enough, I suppose I
can get it set up.

—
Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/1085#issuecomment-145143680.

[au-phiware]

I happily run docker-compose inside a container with these scripts: Powershell scripts for Docker Toolbox for Windows

[dnephin]

I think we can close this issue now.

Master includes support for building a windows binary and support for running compose from an image. Any remaining bugs have their only issues for tracking.

If you run into any problems, please do open an issue about it. Current open issues should be tagged with group/windows: https://github.com/docker/compose/issues?q=is%3Aopen+is%3Aissue+label%3Agroup%2Fwindows

[dnephin]

@au-phiware would you be interested in opening a PR with those scripts? We currently have https://github.com/docker/compose/blob/master/script/run.sh for running the image on linux/osx.

It would be great to include a version that works on windows. It can use the official image we publish (or will be publishing) at docker/compose:<tag>

[atrauzzi]

Just mentioning this here in case people are trying to use bind mounts: #2158

Hopefully it saves you some googling/bug reporting!

[kobenauf]

This was closed over 6 months ago, but volumes still don't mount on Windows (unless they are under c:/users), there is not an easy way to develop on files locally that are then picked up and run by the docker service (unless I am able/willing to move my code to c:/users), and interactive mode is still not supported. Does anyone know if there are specific tickets for those issues I can watch?

[friism]

@kobenauf with beta Docker for Mac and Windows we're trying to address some of these shortcomings: https://blog.docker.com/2016/03/docker-for-mac-windows-beta/

[kobenauf]

Thanks @friism, I'm super happy to hear that! I've submitted my information a couple times for that beta program but never heard back from anybody. Do you happen to know when it might be expanded to more people, and is there any anticipated release date yet?

[airtonix]

docker compose does not properly handle windows environment variables.

example (it's powershell, but I'm using pshazz which changes the prompt to look more unixy):

~ $ echo $SSH_AUTH_SOCK
# blank

~ $ echo $Env:SSH_AUTH_SOCK
/tmp/ssh-qgFBji4276/agent.4276

~ $ gci Env:\SSH_AUTH_SOCK

Name                           Value
----                           -----
SSH_AUTH_SOCK                  /tmp/ssh-qgFBji4276/agent.4276


~ $ echo PWD
PWD
~ $ echo pwd
pwd
~ $ pwd

Path
----
C:\Users\zenobius


~ $ echo $Env:pwd
~ $ echo $Env:PWD
~ $ echo $PWD

Path
----
C:\Users\zenobius


~ $ gci env:pwd
gci : Cannot find path 'pwd' because it does not exist.
At line:1 char:1
+ gci env:pwd
+ ~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (pwd:String) [Get-ChildItem], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand

~ $ gci env:PWD
gci : Cannot find path 'PWD' because it does not exist.
At line:1 char:1
+ gci env:PWD
+ ~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (PWD:String) [Get-ChildItem], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand

~ $

This is preventing me from sharing my $SSH_AUTH_SOCK at runtime into my containers.

[friism]

@airtonix maybe I'm missing something, but I don't see in your example where you invoke docker-compose?

If you have a simple reproducible test, please consider opening a separate issue with details.

[airtonix]

Just assume I am. (Because, indeed I am).

On Tue., 6 Dec. 2016, 02:17 Michael Friis, [e-mail] wrote:

@airtonix https://github.com/airtonix maybe I'm missing something, but
I don't see in your example where you invoke docker-compose?

If you have a simple reproducible test, please consider opening a separate
issue with details.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/1085#issuecomment-264888849,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADvKStX5XgZOG1KJDvXJ-2t5ie14-OCks5rFDH-gaJpZM4DsN6o
.

[dopry]

@airtonix, then please provide an example of exactly what you're doing, so someone else can reproduce it and work on it. Please do it in a new issue so your issue doesn't get lost in the rest of the windows support universe. Please refer to this issue so people interested in windows support issues will be aware. Thanks.