Compose: Provide RPM and DEB packages for compose

[dnephin]

Not everyone likes to install binaries (#1288).

It might be good to provide a system package for compose. We should use https://github.com/spotify/dh-virtualenv for the debian package and look into a similar approach for the rpm.

[jacek99]

I would recommend to just work with the respective Debian and Fedora/CentOS communities and get that rolled into the default system repositories.'

Just my $0.02

[atrauzzi]

That ends up being less than optimal for a project that's moving as quickly as compose. You'll just end up with a pile of people coming in reporting that new features aren't working.

Better to have a PPA so that you can do updates outside of the distro release cycle.

[dnephin]

It sounds like there is a deb for it (https://packages.debian.org/sid/docker-compose) but it's already a few versions behind.

[dnephin]

And a fedora package https://admin.fedoraproject.org/pkgdb/package/docker-compose/

[voltagex]

Couldn't this be done in the same way as https://docs.docker.com/installation/ubuntulinux/ and https://docs.docker.com/installation/fedora/? i.e. host on *.docker.org

[dnephin]

Hosting the packages is the easy part. Most of the work here is making sure the package works for older versions of distros.

[athyuttamre]

Would you consider doing this once development pace has stabilized?

[StefanScherer]

What about putting the docker-compose binary that is also available at the GH release into DEB/RPM?
No python/virtualenv required on the target machine.

Creating such a package with fpm should be very easy. We've done similar things at work (sorry private repos) and put some integration tests behind the build to test the DEB/RPM package at CircleCI within debian/centos Docker containers.

If there is interest in such things I can check if we can open source that parts as an example.

[alexanderadam]

I totally agree that it would nice to have it packaged but for ubuntu I don't think it's useful to add another PPA. I would rather like to see current docker compose in the already existing repository.

@dnephin apt repositories can have different packages per distribution anyway and as the other docker packages are already included it even should be easier to manage that compose and docker work nicely together.

Therefore I think providing automatic builds for the official docker repos should be in focus here (and not creating _additional_ repositories).

[atrauzzi]

@alexanderadam That's a good way to get outdated bug reports. A PPA makes much more sense for a target that moves as quickly as docker+compose.

Indeed that's why people prefer to use the docker PPA. What's in the ubuntu/debian repos just doesn't keep pace.

[mkantor]

@atrauzzi unless I'm misreading, @alexanderadam is suggesting to host docker-compose in the docker repository (https://apt.dockerproject.org/), not rely on the the ubuntu/debian packagers.

[atrauzzi]

Good enough! :)

[ksylvan]

There is also this possibility. Run this shell script as docker-compose:

#!/bin/sh

if [ -t 0 ]; then OPTS='-it'; fi

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
       -v "$(pwd):$(pwd)" -w "$(pwd)" \
       $OPTS dduportal/docker-compose:latest ${1+"$@"}

[FranklinYu]

@ksylvan Why ${1+"$@"} instead of just "$@"?

[Gaff]

Also docker-machine while you're working on it?

[nevergone]

What would there be need for in order for the solution of the issue to go on?

[alexanderadam]

I doubt that anybody from the Docker team had a look on it.

@fsateler, in case you are the packager of the Debian package: is there anything particular to think of when packaging docker-compose as an Debian package (sorry for mentioning).

[fsateler]

@alexanderadam yes, I am the downstream maintainer. I would of course be delighted to have help maintaining the debian package, but if you want to maintain an out-of-band package that is fine too.

If you go the out-of-band way, please consider the following:

1. Packages from such repositories should work with multiple releases (usually debian and ubuntu, in their various releases).
2. This means relying on distro-provided packages is harder. For example, a required version of a python module might not yet be available.
3. Care should be taken to not step over distro-provided packages (that is, overwrite a module with a newer version).

The linked dh-virtualenv above looks like it addresses the above three problems by shipping the complete venv into /opt. All that should be required then is to use the oldest supported disro release to build the packages, this seems a requirement of dh-virtualenv.

The alternative would be to have multiple repositories for the different releases, but that would be IMO much more trouble than it is worth.

PS: yes, the debian package is a bit behind, but that is mostly resolved, hopefully (we have now 1.8.0). We were stuck a while waiting for newer versions of some dependent modules.

[gyszalai]

I think providing a snap or flatpak package could solve the dependency issues. Opinions?

[jasonmp85]

So why isn't there already at least just a package with the binary in it? After poking around how exactly this binary is generated, it appears it's dynamically linked, so I guess there are concerns about breaking compatibility with older distros, but right now your "upgrade" steps are basically "blast over the old binary and hope it works", so I don't particularly see that as any sort of regression (i.e. the dependencies aren't specified as-is, and neither are compatible distros, so slapping the binary into a package would be no worse than the current scenario, and in some ways actually better, since it permits users to install docker and compose in one line rather than two separate ways).

I guess compose is kind of a dev tool in the first place, but it would be nice to see a better install story.

[alexanderadam]

So why isn't there already at least just a package with the binary in it?

@jasonmp85 just to clear this up: as mentioned earlier there are actually packages and fsateler (maintainer of the current Debian packages) wrote that he would be more than happy if someone could support him maintaining these packages. And if you _don't need_ the latest Docker binaries you will be probably fine if you just use them together with the Docker binaries provided by your Distro (and you get security updates via your package manager as well).

However, there are a bunch of good reasons to prefer the _current_ version of Docker. Which was most probably the motivation why the Docker people created a PPA and wrote that you should use it. But docker-compose is missing here.

PS: I don't quite understand why issue 24210 was closed. I agree that there is tooling required first (so this issue here is a dependency for 24210) but this shouldn't be a reason to _close_ this issue, right?

[FranklinYu]

Note: the Debian package is already in stretch, the next release, and it has been in Ubuntu since Xenial. Being easy to update is more important than cool new functionality to me, so it being un-APT-able stops me from using the latest version.

update

I switch to installing with pip within a venv, and aliasing docker-compose to the binary within that venv:

alias docker-compose=~/.python-global-venv/bin/docker-compose

I think this is the best solution by far; it depends only on python3-venv.

[kfiresmith]

+1 for a Compose RPM in the Docker Inc CentOS repos.

As a new user of Docker CE I had assumed that compose was in the main RPM when I saw that there were only the two rpms (docker-ce, docker-ce-selinux) in the CentOS repo. Had to google around to find this page and get caught up on the state of this.

[carlwgeorge]

docker-compose is now available in the EPEL repository for RHEL/CentOS 7. I was limited to version 1.9 because Red Hat only ships the Python SDK 1, and newer versions require SDK 2. If at some point in the future Red Hat updates that to SDK 2, I can update docker-compose to a newer version. For now, that version is compatible with compose file format 2.2, which is compatible with both the docker 1.12.6 and docker-latest 1.13.1 packages from the distribution.

[Toub]

Would be nice to add compose to docker-ce in official debian repository: https://download.docker.com/linux/debian

[debrouxl]

I also think it would be nice to add a docker-compose package into the official Debian repository :)

[francislavoie]

As @Toub said, I wish compose was included with docker-ce. I went looking here: https://download.docker.com/linux/ubuntu/dists/xenial/stable/ to see if it was included, and it still isn't. The release notes say for Windows and Mac, Compose is included. Why not Linux? It's about time Docker, the _Linux_ tool, gets proper package updates for people running this on servers. It's stupid to have to maintain a separate script to keep Compose versions in line on production servers.

[ksylvan]

I just use the shell script which runs docker-compose in a container.

Just do:

1. Copy https://github.com/docker/compose/releases/download/1.15.0/run.sh to /usr/local/bin/docker-compose
2. Copy https://raw.githubusercontent.com/docker/compose/1.15.0/contrib/completion/bash/docker-compose to /etc/bash_completions.d/docker-compose for the bash completions

[FranklinYu]

@ksylvan That's more like a workaround than a solution…

[alexanderadam]

@ksylvan the idea of package managers is that you are able to manage (ie. upgrade) your packages at once and having dependencies that match to each other. _Manual_ solutions (including yours) won't be able to enable this.

@franklinyu your solution isn't much more better. Relying on the pip packages requires users to having to manage and update the packages from a second package manager. Furthermore it won't guarantee that the pip version is compatible to your installed docker version and paths/dependency from your distribution.

[hackel]

Docker-compose desperately needs an official snap package. It's cross-platform, encapsulates all dependencies, gives you automatic upgrades, and is just set it and forget it. As a developer, having to remember to come manually check if any new versions have been released and manually install them is just silly. Maintaining separate deb/rpm/whatever packages for each version of a distribution is extra work no one needs.

[kfiresmith]

Maintaining separate deb/rpm/whatever packages for each version of a distribution is extra work no one needs.

There are lots of operating environments that cannot use a 3rd party package manager such as Snap due to policy controls. Without going into too much detail, think GOVT sites, etc...
So while snaps are great for independent devs and other freewheeling environments, they are not an actual replacement to OS-native/Distro-native packages such as .deb/.rpm.

[sweepyoface]

Maintaining separate deb/rpm/whatever packages for each version of a distribution is extra work no one needs.

There are lots of operating environments that cannot use a 3rd party package manager such as Snap due to policy controls. Without going into too much detail, think GOVT sites, etc...
So while snaps are great for independent devs and other freewheeling environments, they are not an actual replacement to OS-native/Distro-native packages such as .deb/.rpm.

How about both? :stuck_out_tongue:

---

For anyone looking to use this:

There is also this possibility. Run this shell script as docker-compose:

#!/bin/sh

if [ -t 0 ]; then OPTS='-it'; fi

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
       -v "$(pwd):$(pwd)" -w "$(pwd)" \
       $OPTS dduportal/docker-compose:latest ${1+"$@"}

the dduportal/docker-compose image hasn't been updated since 1.12.0, and there is now an official docker/compose image.

Use this instead (requires jq to be installed, or alias jq to also run with docker):

#!/bin/sh

if [ -t 0 ]; then OPTS='-it'; fi

latest=$(curl -s https://registry.hub.docker.com/v2/repositories/docker/compose/tags/ | jq -r '.results[0].name')

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
       -v "$(pwd):$(pwd)" -w "$(pwd)" \
       $OPTS docker/compose:"$latest" ${1+"$@"}

Note that this includes release candidate versions.

The following script does not include prereleases, but fetches the release from GitHub instead of directly from the Docker Hub (so it's possible the tag won't exist on the Docker Hub):

#!/bin/sh

if [ -t 0 ]; then OPTS='-it'; fi

latest=$(curl -s https://api.github.com/repos/docker/compose/releases/latest | jq -r '.tag_name')

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
       -v "$(pwd):$(pwd)" -w "$(pwd)" \
       $OPTS docker/compose:"$latest" ${1+"$@"}

This isn't ideal since there is no latest tag (until https://github.com/docker/compose/pull/6553) and the API request will take extra time, but it works.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[FranklinYu]

I would say let's close it since Docker team clearly isn't interested given that we don't hear from them for more than 3 years. Community cannot do anything because we don't control the repository. We have enough workaround above.

[stale[bot]]

This issue has been automatically marked as not stale anymore due to the recent activity.

[cefn]

For reference, following the most recent suggestion from @sweepyoface in https://github.com/docker/compose/issues/2235#issuecomment-477158800 on my system at least, access to /var/run/docker.sock requires sudo

Without prefixing the docker-compose command with sudo...

$ /home/cefn/.local/bin/docker-compose --help
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.

Update: responding to @oxydaemon's comment below:

'You need to use sudo to run Docker commands.'
https://docs.docker.com/install/linux/docker-ce/debian/

The information at https://docs.docker.com/install/linux/linux-postinstall/ shows how to manage access to docker through groups (instead of sudo). However note 'The docker group grants privileges equivalent to the root user' ( https://docs.docker.com/install/linux/linux-postinstall/ ) so until this is mitigated I'll personally keep using sudo.

[oxydaemon]

For reference, following the most recent suggestion from @sweepyoface in #2235 (comment) on my system at least, access to /var/run/docker.sock requires sudo

Without prefixing the docker-compose command with sudo...

$ /home/cefn/.local/bin/docker-compose --help
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.

You shouldn't need sudo, that is permission denied error to docker.sock.
Assuming you're running as user 'cefn', did you add that user to the docker group in /etc/group?

username@myserver:~$ ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Feb 27 14:33 /var/run/docker.sock

docker group members have read and write permissions

[cefn]

I switch to installing with pip within a venv, and aliasing docker-compose to the binary within that venv ... it depends only on python3-venv.

The alternative https://docs.docker.com/compose/install/ under 'Alternative Install Options' is even simpler, and doesn't rely on python3-venv either...

sudo pip install docker-compose

Unfortunately 'Alternative Install Options' is set against tabs for 'Windows, Mac OS, Linux'. The Linux tab contains only curl-based instructions, (which swing free from software repositories or updates).

I'd personally like to see 'install Pip' then 'pip install docker-compose' under the Linux tab. However, perhaps hiding the pip approach as an 'Alternative Install' is to avoid having to provide multi-os guidance how to install pip.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HMSCelestia]

Unstale this.

[stale[bot]]

This issue has been automatically marked as not stale anymore due to the recent activity.